You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by wu...@apache.org on 2026/05/24 15:56:03 UTC
(skywalking) branch master updated: Runtime MAL/LAL rules can declare dynamic layers (#13883)
This is an automated email from the ASF dual-hosted git repository.
wu-sheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/skywalking.git
The following commit(s) were added to refs/heads/master by this push:
new 40d95a179b Runtime MAL/LAL rules can declare dynamic layers (#13883)
40d95a179b is described below
commit 40d95a179b0af6fab342316bce905f28918dd4ee
Author: 吴晟 Wu Sheng <wu...@foxmail.com>
AuthorDate: Sun May 24 23:55:57 2026 +0800
Runtime MAL/LAL rules can declare dynamic layers (#13883)
* Runtime MAL/LAL rules can declare dynamic layers
Runtime hot-update rules pushed through the runtime-rule admin API may
now declare a layerDefinitions: block. Layers register through a new
Layer.registerDynamic channel (post-seal-allowed) and are refcount-
tracked by RuntimeLayerRegistry: removing the last declaring rule
unregisters the layer. Ordinals are operator-pinned in the 100_000+
tier (no auto-allocation; the ordinal is persisted in ServiceTraffic
primary keys and must be operator-stable across restarts).
Bundled and runtime are strictly separate ownership channels:
* Runtime overrides of bundled rules cannot carry layerDefinitions —
REST rejects with applyStatus=layer_override_forbidden.
* Legacy override rows on disk-twins drop the static-loader
substitution and apply post-seal via the dynamic channel so they
remain operator-removable.
* RuleSetMerger no longer injects resolver-only ACTIVE rows into the
static loader; pure-runtime rules go through RuleSync.runOnce.
Conflict validation surfaces structured 400s with new applyStatus
codes: layer_ordinal_out_of_range, layer_name_conflict,
layer_ordinal_collision, layer_name_invalid, layer_override_forbidden.
The checker covers in-batch duplicates, self-update multi-claimant
ordinal reuse, and bundled-vs-runtime conflicts.
E2E (test/e2e-v2/cases/runtime-rule/mal-storage/runtime-rule-flow.sh)
extended with Phase 5e-5h: three layer-rejection assertions plus a
restart-survival round-trip that POST -> swctl layer ls -> docker
restart -> swctl layer ls still shows -> /delete -> swctl layer ls
no longer shows.
See docs/en/concepts-and-designs/runtime-rule-hot-update.md#dynamic-layers
for the full contract.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Replace bare assert with assertTrue in LayerDynamicRegisterTest
Address Copilot review on PR #13883. Project's surefire enableAssertions
defaults to true so bare `assert` does fire, but JUnit assertions are
the convention used by every other check in these tests and survive any
test-runner configuration.
---
CLAUDE.md | 20 ++
docs/en/changes/changes.md | 5 +
.../runtime-rule-hot-update.md | 83 +++++
.../oap/log/analyzer/v2/provider/LALConfigs.java | 7 +-
.../provider/LALConfigsLayerDefinitionsTest.java | 1 +
.../meter/analyzer/v2/prometheus/rule/Rules.java | 7 +-
.../prometheus/rule/RulesLayerDefinitionsTest.java | 1 +
.../runtimerule/apply/DeltaClassifier.java | 132 ++++++-
.../receiver/runtimerule/apply/LalFileApplier.java | 114 +++++-
.../receiver/runtimerule/apply/MalFileApplier.java | 108 +++++-
.../runtimerule/engine/lal/LalRuleEngine.java | 14 +
.../runtimerule/engine/mal/MalRuleEngine.java | 15 +
.../extension/DbOverrideRuntimeRuleResolver.java | 75 +++-
.../receiver/runtimerule/layer/AppliedClaims.java | 65 ++++
.../receiver/runtimerule/layer/LayerClaim.java | 74 ++++
.../runtimerule/layer/LayerConflictException.java | 73 ++++
.../layer/RuntimeLayerConflictChecker.java | 290 ++++++++++++++++
.../runtimerule/layer/RuntimeLayerRegistry.java | 385 +++++++++++++++++++++
.../runtimerule/reconcile/DSLRuntimeApply.java | 6 +
.../runtimerule/rest/RuntimeRuleService.java | 66 ++++
.../receiver/runtimerule/state/EngineApplied.java | 17 +
.../runtimerule/apply/MalFileApplierTest.java | 81 +++++
.../DbOverrideRuntimeRuleResolverTest.java | 116 +++++++
.../layer/RuntimeLayerRegistryTest.java | 383 ++++++++++++++++++++
.../skywalking/oap/server/core/analysis/Layer.java | 201 ++++++++++-
.../oap/server/core/rule/ext/RuleSetMerger.java | 9 +-
.../core/rule/ext/RuntimeRuleOverrideResolver.java | 10 +-
.../core/analysis/LayerDynamicRegisterTest.java | 196 +++++++++++
.../src/main/resources/layer-extensions.yml | 47 ++-
.../runtime-rule/mal-storage/banyandb/e2e.yaml | 7 +-
.../runtime-rule/mal-storage/runtime-rule-flow.sh | 113 +++++-
.../seed-rules/illegal-layer-name-conflict.yaml | 29 ++
.../seed-rules/illegal-layer-name-invalid.yaml | 27 ++
.../seed-rules/illegal-layer-out-of-range.yaml | 32 ++
.../seed-rules/seed-rule-sibling-with-layer.yaml | 33 ++
35 files changed, 2758 insertions(+), 84 deletions(-)
diff --git a/CLAUDE.md b/CLAUDE.md
index 1608fda657..02742e1f60 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -155,6 +155,26 @@ public class XxxModuleProvider extends ModuleProvider {
- Use Lombok annotations (`@Getter`, `@Setter`, `@Builder`, `@Data`, `@Slf4j`, etc.)
- Follow existing patterns in similar files
+**Comments:**
+- Don't explain what the code already says. Comment only when reading the code is
+ not enough — invariants that aren't visible in this method, cross-cutting
+ ordering, why a non-obvious branch exists, why a seemingly redundant call is
+ required. If the code is clear, no comment.
+- Connect logic that lives in different places. A check here that exists only
+ because of a guarantee made elsewhere deserves a one-line pointer to that
+ guarantee. A workaround for a known bug elsewhere deserves a link.
+- Trim restated javadoc. Method-level docs should describe the contract, not
+ re-narrate the body. Multi-paragraph rationale belongs in the design doc, not
+ the source.
+
+**Tests:**
+- Always use JUnit assertions (`assertTrue`, `assertEquals`, `assertThrows`, …).
+ Never write bare Java `assert` statements in tests. Although surefire defaults
+ `enableAssertions=true` so bare `assert` does fire in this project's tests,
+ JUnit assertions are the convention everywhere else in the codebase, survive
+ any test-runner configuration, and produce a structured failure that names
+ the actual value instead of a context-free `AssertionError`.
+
### License Header
Java, XML, and YAML/YML files must include the Apache 2.0 license header (see `HEADER` file).
JSON and Markdown files are excluded (JSON doesn't support comments, see `.licenserc.yaml`).
diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md
index b2804f1359..604b6be4e9 100644
--- a/docs/en/changes/changes.md
+++ b/docs/en/changes/changes.md
@@ -238,6 +238,11 @@
admin-host only" entry above for the public REST retirement.
#### OAP Server
+* Runtime MAL/LAL hot-update rules can declare `layerDefinitions:` to introduce new
+ layers. Ordinals are operator-pinned in the `100_000+` tier; the layer is
+ refcount-tracked and unregistered when the last declaring rule is removed. See
+ [runtime-rule-hot-update.md#dynamic-layers](../concepts-and-designs/runtime-rule-hot-update.md)
+ for the conflict rules and limitations.
* Fix: remove the redundant tags from the `envoy-ai-gateway.yaml` LAL configuration.
* Add Zipkin Virtual GenAI e2e test. Use `zipkin_json` exporter to avoid protobuf dependency conflict
between `opentelemetry-exporter-zipkin-proto-http` (protobuf~=3.12) and `opentelemetry-proto` (protobuf>=5.0).
diff --git a/docs/en/concepts-and-designs/runtime-rule-hot-update.md b/docs/en/concepts-and-designs/runtime-rule-hot-update.md
index 78ee77f6d4..4abf49f7dd 100644
--- a/docs/en/concepts-and-designs/runtime-rule-hot-update.md
+++ b/docs/en/concepts-and-designs/runtime-rule-hot-update.md
@@ -432,6 +432,89 @@ status, per-node `localState`, and `lastApplyError` for any rule whose most rece
apply failed. There is no separate alert channel — `/list` plus the OAP log are
the entire diagnostic surface.
+## Dynamic layers
+
+Runtime MAL/LAL rules MAY introduce new layer names through the same
+`layerDefinitions:` block bundled rule files use:
+
+```yaml
+layerDefinitions:
+ - name: MY_NEW_LAYER
+ ordinal: 100050 # required, runtime tier (>=100_000)
+ # normal: true # optional; defaults to true
+metricPrefix: my_prefix
+metricsRules:
+ - name: requests
+ exp: any.sum(['service', 'instance'])
+```
+
+### Ordinal tier — operator-pinned
+
+| Range | Channel |
+|------------------------------|------------------------------------------------------------------------|
+| `0 – 9_999` | Built-in `Layer.*` constants |
+| `10_000 – 99_999` | `layer-extensions.yml` + bundled MAL/LAL `layerDefinitions:` |
+| `100_000 – Integer.MAX_VALUE`| Runtime DSL dynamic layers |
+
+The OAP does NOT auto-allocate. Ordinals land in persisted `ServiceTraffic` primary
+keys; they must be operator-stable across restarts.
+
+### Lifecycle
+
+`RuntimeLayerRegistry` refcounts each runtime claim per declaring rule. When the last
+runtime claim is removed (`/delete` of the rule, or `/addOrUpdate` that drops the
+entry), the layer is unregistered if it was originally registered through the runtime
+channel.
+
+### Limitations
+
+- **Runtime overrides of bundled rules cannot declare `layerDefinitions:`**. Bundled
+ and runtime are separate layer-ownership channels — never overwrite, never share. The
+ REST handler rejects `/addOrUpdate` of a rule whose name matches a bundled disk file
+ AND whose body carries a non-empty `layerDefinitions:` block with `applyStatus =
+ layer_override_forbidden` (HTTP 400). Operators wanting new layers on a bundled rule
+ must either edit the bundled source on disk and restart, OR push a pure-runtime rule
+ with a different name that declares the layer.
+- **Legacy override rows** (persisted before the rejection was introduced) are handled
+ at boot: the static-loader substitution is dropped so the bundled disk content loads
+ with its original layers, and `RuleSync` then applies the runtime row dynamically
+ post-seal — the runtime row's `layerDefinitions` go through the dynamic channel and
+ are operator-removable via `/inactivate` + `/delete`.
+- **Pure runtime rules (no bundled twin) ARE fully removable**. The layer registers
+ through the runtime channel and `unregisterDynamic` succeeds when the refcount hits
+ zero.
+- **Removing a layer with historical `ServiceTraffic` rows orphans those rows** — the
+ persisted ordinal no longer resolves and reads throw `Unknown Layer value`. Operator
+ must migrate or purge the data before removal; the feature does not auto-clean.
+- **No redeclaration of built-in or boot-time-external layers** at runtime. Use the
+ existing name in your rule body (`layer: GENERAL`, `dest: ...Layer.MESH...`) — do
+ not list those names under `layerDefinitions:`.
+
+### Conflict rules
+
+Validation runs before any apply lands and returns HTTP 400 with the standard
+`{applyStatus, catalog, name, message}` envelope.
+
+| `applyStatus` | Trigger |
+|------------------------------|------------------------------------------------------------------------------------------------------------|
+| `layer_ordinal_out_of_range` | `ordinal:` missing (default 0) or `< 100_000`. |
+| `layer_name_invalid` | `name` does not match `[A-Z][A-Z0-9_]*`. |
+| `layer_name_conflict` | Same `name` already registered with a different `(ordinal, normal)` by another rule (bundled or runtime). Also raised for same-batch duplicate names with different triples, and for self-edit triple changes when another runtime rule shares the layer. |
+| `layer_ordinal_collision` | Different `name`, same `ordinal` already in use. Also raised for same-batch duplicate ordinals, and for self-edit ordinal reuse when another runtime rule still holds the prior layer at that ordinal. |
+| `layer_override_forbidden` | `/addOrUpdate` on a rule with a bundled disk twin where the body contains `layerDefinitions:`. Bundled rules own their layer declarations; runtime overrides may only change the rule body. |
+
+Same triple as an existing bundled layer is permitted as a soft claim (see Limitations
+above). Same triple as an existing runtime layer is a no-op refcount-add. Self-edits
+(changing the triple of a layer this rule already owns) are permitted only when the
+rule is the sole claimant; if other rules share the name, the operator must align both
+sides in lockstep or remove the prior declaration first.
+
+INACTIVE rules hold no layer claims — their refcount entries are dropped at
+`/inactivate`. Reactivation (`/addOrUpdate` against an INACTIVE row) runs full
+validation; if another rule started claiming the name in the meantime,
+`layer_name_conflict` surfaces and the operator must edit or `/delete` the inactive
+rule before it can come back.
+
## What this feature does not do
- **OAL hot-update** is out of scope (see "Scope" above).
diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/provider/LALConfigs.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/provider/LALConfigs.java
index f0dea01585..04f73cf484 100644
--- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/provider/LALConfigs.java
+++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/provider/LALConfigs.java
@@ -71,12 +71,13 @@ public class LALConfigs {
*
* <p>Compared with the legacy disk-only path:
* <ul>
- * <li>Files in {@code files} but missing on disk are still loaded if a resolver
- * contributes ACTIVE content for them (DB-only LAL rules).</li>
* <li>Files on disk + in allow-list with an INACTIVE resolver entry are skipped.</li>
* <li>Files on disk + in allow-list with an ACTIVE resolver entry are parsed from
* resolver bytes (override).</li>
* <li>Files on disk + in allow-list with no resolver opinion are parsed from disk.</li>
+ * <li>Files in resolver contributions but not on disk are NOT loaded here — pure
+ * runtime LAL rules go through {@code RuleSync.runOnce} post-seal via the
+ * dynamic layer channel, which preserves operator-removable ownership.</li>
* </ul>
*/
public static List<LALConfigs> load(final String path, final List<String> files,
@@ -155,7 +156,7 @@ public class LALConfigs {
* Funnel any inline {@code layerDefinitions:} entries through {@code Layer.register}.
* Conflict checks (reserved-range, name uniqueness, ordinal uniqueness, sealed-state) live
* in {@code Layer.register}; failures here surface with the offending rule name in
- * the stack trace, which is enough for an operator to find the bad file.
+ * the stack trace.
*/
private static void registerInlineLayers(final String ruleName, final LALConfigs configs) {
final List<LayerDefinition> defs = configs.getLayerDefinitions();
diff --git a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/provider/LALConfigsLayerDefinitionsTest.java b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/provider/LALConfigsLayerDefinitionsTest.java
index dbf45b6028..5edee31448 100644
--- a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/provider/LALConfigsLayerDefinitionsTest.java
+++ b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/provider/LALConfigsLayerDefinitionsTest.java
@@ -72,4 +72,5 @@ class LALConfigsLayerDefinitionsTest {
assertEquals(1300, Layer.nameOf("TEST_LAL_LAYER_A").value());
}
+
}
diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/prometheus/rule/Rules.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/prometheus/rule/Rules.java
index 5be3b570a9..4b0e369e1e 100644
--- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/prometheus/rule/Rules.java
+++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/prometheus/rule/Rules.java
@@ -131,8 +131,9 @@ public class Rules {
}
// Merge with classpath-discovered resolvers (runtime-rule DB, plus any future
- // priority-ranked source). Resolvers contributing INACTIVE drop their entries;
- // ACTIVE substitutes content. Resolver-only rules (not on disk) are included.
+ // priority-ranked source). ACTIVE substitutes existing disk entries; INACTIVE
+ // drops them. Resolver-only rules (no disk twin) are NOT merged here — they're
+ // applied by RuleSync.runOnce post-seal via the dynamic layer channel.
final Map<String, byte[]> merged = useInstalledManager
? RuleSetMerger.merge(path, diskBytes)
: RuleSetMerger.merge(path, diskBytes, manager);
@@ -161,7 +162,7 @@ public class Rules {
* Funnel any inline {@code layerDefinitions:} entries through {@code Layer.register}.
* Conflict checks (reserved-range, name uniqueness, ordinal uniqueness, sealed-state) live
* in {@code Layer.register}; failures here surface with the offending rule name in
- * the stack trace, which is enough for an operator to find the bad file.
+ * the stack trace.
*/
private static void registerInlineLayers(final String ruleName, final Rule rule) {
final List<LayerDefinition> defs = rule.getLayerDefinitions();
diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/v2/prometheus/rule/RulesLayerDefinitionsTest.java b/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/v2/prometheus/rule/RulesLayerDefinitionsTest.java
index 55811cdf36..0d431845cf 100644
--- a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/v2/prometheus/rule/RulesLayerDefinitionsTest.java
+++ b/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/v2/prometheus/rule/RulesLayerDefinitionsTest.java
@@ -74,4 +74,5 @@ class RulesLayerDefinitionsTest {
assertEquals(1200, Layer.nameOf("TEST_MAL_LAYER_A").value());
}
+
}
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/DeltaClassifier.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/DeltaClassifier.java
index 5548bfba92..6d0e4b42f7 100644
--- a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/DeltaClassifier.java
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/DeltaClassifier.java
@@ -131,19 +131,84 @@ public final class DeltaClassifier {
}
}
- if (added.isEmpty() && removed.isEmpty() && shapeBreak.isEmpty()) {
- // Same metric-name set, same shape for every one. Safe to skip DDL, alarm reset,
- // and L1/L2 drain — the fast path.
- return DSLDelta.filterOnly("body/filter/tag edits only (shapes unchanged)");
+ // layerDefinitions diff escalates to STRUCTURAL even when metric shape is unchanged.
+ // The runtime-layer registry must validate the change atomically with the apply, and
+ // the FILTER_ONLY fast path persists BEFORE apply — a layer conflict surfacing
+ // post-persist would leak a row that subsequently fails to register. STRUCTURAL
+ // runs apply first, so any LayerConflictException surfaces before persist.
+ final String layerDelta = describeMalLayerDelta(oldContent, newContent);
+
+ if (added.isEmpty() && removed.isEmpty() && shapeBreak.isEmpty() && layerDelta == null) {
+ // Same metric-name set, same shape for every one, layerDefinitions unchanged.
+ // Safe to skip DDL, alarm reset, and L1/L2 drain — the fast path.
+ return DSLDelta.filterOnly("body/filter/tag edits only (shapes + layers unchanged)");
}
final Set<String> shapeBreakFrozen = shapeBreak.isEmpty()
? Collections.emptySet()
: Collections.unmodifiableSet(shapeBreak);
- final String reason = reasonFor(added, removed, shapeBreakFrozen);
+ final String reason = layerDelta != null
+ ? reasonFor(added, removed, shapeBreakFrozen) + "; " + layerDelta
+ : reasonFor(added, removed, shapeBreakFrozen);
return DSLDelta.structural(added, removed, shapeBreakFrozen, reason);
}
+ /**
+ * Describe the layer-definitions diff between two MAL YAML strings, or {@code null} when
+ * the lists are identical (or both absent). Both lists are read via the same SnakeYAML
+ * binding the applier uses; a parse failure on either side returns a conservative
+ * "layer parse failed" so the caller escalates rather than risking a partial apply.
+ */
+ private static String describeMalLayerDelta(final String oldContent, final String newContent) {
+ final Set<String> oldLayers = safeLayerSignatures(oldContent);
+ final Set<String> newLayers = safeLayerSignatures(newContent);
+ if (oldLayers.equals(newLayers)) {
+ return null;
+ }
+ final Set<String> added = new LinkedHashSet<>(newLayers);
+ added.removeAll(oldLayers);
+ final Set<String> removed = new LinkedHashSet<>(oldLayers);
+ removed.removeAll(newLayers);
+ if (added.isEmpty() && removed.isEmpty()) {
+ return null;
+ }
+ final StringBuilder sb = new StringBuilder("layerDefinitions changed");
+ if (!added.isEmpty()) {
+ sb.append(" (added=").append(added).append(")");
+ }
+ if (!removed.isEmpty()) {
+ sb.append(" (removed=").append(removed).append(")");
+ }
+ return sb.toString();
+ }
+
+ /** Sticker for a single {@code LayerDefinition} encoding {@code (name, ordinal, normal)}.
+ * Two stickers are equal iff every field matches — i.e., idempotent re-declarations of
+ * the same triple produce identical strings and are correctly treated as no-op. */
+ private static Set<String> safeLayerSignatures(final String content) {
+ if (content == null || content.isEmpty()) {
+ return Collections.emptySet();
+ }
+ try (StringReader reader = new StringReader(content)) {
+ final Rule rule = new Yaml().loadAs(reader, Rule.class);
+ if (rule == null || rule.getLayerDefinitions() == null
+ || rule.getLayerDefinitions().isEmpty()) {
+ return Collections.emptySet();
+ }
+ final Set<String> out = new LinkedHashSet<>();
+ for (final org.apache.skywalking.oap.server.core.analysis.LayerDefinition d
+ : rule.getLayerDefinitions()) {
+ out.add(d.getName() + ":" + d.getOrdinal() + ":" + d.isNormal());
+ }
+ return out;
+ } catch (final RuntimeException ex) {
+ // Conservative on parse failure: caller treats non-null as a change, so reporting
+ // an empty set when one side fails and a non-empty otherwise still escalates.
+ // Return a sentinel so equals() reports a diff regardless of the other side.
+ return Collections.singleton("__layer_parse_failed__:" + ex.getMessage());
+ }
+ }
+
/**
* Classify a LAL rule-file delta. LAL has no direct metric-name target for alarm windows
* (rule keys are {@code (layer, ruleName)} pairs, not metric names), so the added/removed/
@@ -170,9 +235,62 @@ public final class DeltaClassifier {
// prior windows existed).
return DSLDelta.newRule(Collections.emptySet());
}
+ // Layer-definition diffs are surfaced in the reason string so operators can see why
+ // a FILTER_ONLY-shaped edit was classified STRUCTURAL. LAL currently classifies all
+ // content changes as STRUCTURAL anyway (no per-rule shape extraction is implemented),
+ // so the layer diff just enriches the reason rather than changing the verdict.
+ final String layerDelta = describeLalLayerDelta(oldContent, newContent);
+ final String reason = layerDelta != null
+ ? "LAL content changed; " + layerDelta
+ : "LAL content changed";
return DSLDelta.structural(
- Collections.emptySet(), Collections.emptySet(), Collections.emptySet(),
- "LAL content changed");
+ Collections.emptySet(), Collections.emptySet(), Collections.emptySet(), reason);
+ }
+
+ /** LAL variant of {@link #describeMalLayerDelta} — reads from {@link LALConfigs}
+ * instead of {@link Rule}. Identical semantics. */
+ private static String describeLalLayerDelta(final String oldContent, final String newContent) {
+ final Set<String> oldLayers = safeLalLayerSignatures(oldContent);
+ final Set<String> newLayers = safeLalLayerSignatures(newContent);
+ if (oldLayers.equals(newLayers)) {
+ return null;
+ }
+ final Set<String> added = new LinkedHashSet<>(newLayers);
+ added.removeAll(oldLayers);
+ final Set<String> removed = new LinkedHashSet<>(oldLayers);
+ removed.removeAll(newLayers);
+ if (added.isEmpty() && removed.isEmpty()) {
+ return null;
+ }
+ final StringBuilder sb = new StringBuilder("layerDefinitions changed");
+ if (!added.isEmpty()) {
+ sb.append(" (added=").append(added).append(")");
+ }
+ if (!removed.isEmpty()) {
+ sb.append(" (removed=").append(removed).append(")");
+ }
+ return sb.toString();
+ }
+
+ private static Set<String> safeLalLayerSignatures(final String content) {
+ if (content == null || content.isEmpty()) {
+ return Collections.emptySet();
+ }
+ try (StringReader reader = new StringReader(content)) {
+ final LALConfigs configs = new Yaml().loadAs(reader, LALConfigs.class);
+ if (configs == null || configs.getLayerDefinitions() == null
+ || configs.getLayerDefinitions().isEmpty()) {
+ return Collections.emptySet();
+ }
+ final Set<String> out = new LinkedHashSet<>();
+ for (final org.apache.skywalking.oap.server.core.analysis.LayerDefinition d
+ : configs.getLayerDefinitions()) {
+ out.add(d.getName() + ":" + d.getOrdinal() + ":" + d.isNormal());
+ }
+ return out;
+ } catch (final RuntimeException ex) {
+ return Collections.singleton("__layer_parse_failed__:" + ex.getMessage());
+ }
}
/**
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/LalFileApplier.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/LalFileApplier.java
index cdca5ca236..415e81172a 100644
--- a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/LalFileApplier.java
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/LalFileApplier.java
@@ -31,12 +31,16 @@ import lombok.extern.slf4j.Slf4j;
import org.apache.skywalking.oap.log.analyzer.v2.module.LogAnalyzerModule;
import org.apache.skywalking.oap.log.analyzer.v2.provider.LALConfig;
import org.apache.skywalking.oap.server.core.analysis.Layer;
+import org.apache.skywalking.oap.server.core.analysis.LayerDefinition;
import org.apache.skywalking.oap.log.analyzer.v2.provider.LALConfigs;
import org.apache.skywalking.oap.log.analyzer.v2.provider.log.listener.LogFilterListener;
import org.apache.skywalking.oap.server.core.classloader.Catalog;
import org.apache.skywalking.oap.server.core.classloader.DSLClassLoaderManager;
import org.apache.skywalking.oap.server.core.classloader.RuleClassLoader;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.AppliedClaims;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.LayerClaim;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.RuntimeLayerRegistry;
import org.apache.skywalking.oap.server.receiver.runtimerule.state.EngineApplied;
import org.yaml.snakeyaml.Yaml;
@@ -54,9 +58,17 @@ import org.yaml.snakeyaml.Yaml;
public class LalFileApplier {
private final LogFilterListener.Factory factory;
+ private final RuntimeLayerRegistry layerRegistry;
public LalFileApplier(final LogFilterListener.Factory factory) {
+ this(factory, RuntimeLayerRegistry.INSTANCE);
+ }
+
+ /** Test-friendly constructor — pass a dedicated registry to isolate from JVM-wide state. */
+ public LalFileApplier(final LogFilterListener.Factory factory,
+ final RuntimeLayerRegistry layerRegistry) {
this.factory = factory;
+ this.layerRegistry = layerRegistry;
}
/**
@@ -69,9 +81,10 @@ public class LalFileApplier {
* later inside {@link #apply}).
*/
public List<RegisteredRule> planKeys(final String yamlContent, final String sourceName) throws ApplyException {
- final List<LALConfig> configs = parse(yamlContent, sourceName);
- final List<RegisteredRule> keys = new ArrayList<>(configs.size());
- for (final LALConfig c : configs) {
+ final LALConfigs configs = parse(yamlContent, sourceName);
+ final List<LALConfig> rules = configs.getRules();
+ final List<RegisteredRule> keys = new ArrayList<>(rules.size());
+ for (final LALConfig c : rules) {
final boolean isAuto = LALConfig.LAYER_AUTO.equalsIgnoreCase(c.getLayer());
final Layer layer = isAuto
? null
@@ -116,7 +129,18 @@ public class LalFileApplier {
public Applied apply(final String yamlContent, final String sourceName,
final String contentHash,
final DSLClassLoaderManager.Kind kind) throws ApplyException {
- final List<LALConfig> configs = parse(yamlContent, sourceName);
+ final LALConfigs configs = parse(yamlContent, sourceName);
+ final List<LALConfig> rules = configs.getRules();
+ final List<LayerClaim> layerClaims = extractLayerClaims(configs);
+
+ // Register any declared layers FIRST so the LAL `layer:` field on rules in this file
+ // can resolve against the freshly-introduced names before Phase 1 compile starts.
+ // Conflicts throw LayerConflictException (REST handler translates to HTTP 400 with
+ // the structured envelope); atomic — validate() rejects before any registration.
+ final String catalogStr = deriveCatalog(sourceName);
+ final String ruleNameStr = deriveRuleName(sourceName);
+ final AppliedClaims appliedClaims = layerRegistry.apply(
+ RuntimeLayerRegistry.ruleId(catalogStr, ruleNameStr), layerClaims);
// One per-file RuleClassLoader for the whole file — every rule inside shares it, so all
// generated LalExpression classes (one per rule) drop together on unregister when the
@@ -148,13 +172,16 @@ public class LalFileApplier {
// volatile map write; a partial-failure window here is theoretical (Map.put
// doesn't throw). We still track progress per-rule so if somehow the JVM throws
// during phase 2, the caller's rollback list is accurate.
- final List<LogFilterListener.Factory.CompiledLAL> compiled = new ArrayList<>(configs.size());
- for (final LALConfig c : configs) {
+ final List<LogFilterListener.Factory.CompiledLAL> compiled = new ArrayList<>(rules.size());
+ for (final LALConfig c : rules) {
c.setSourceName(sourceName);
try {
compiled.add(factory.compile(c, pool, ruleLoader));
} catch (final Throwable t) {
// Compile-phase failure: zero registrations landed, so partial is empty.
+ // Roll back the layer registrations done above so a failed compile does not
+ // leak runtime-layer state.
+ layerRegistry.rollback(appliedClaims);
throw new ApplyException(
"LAL compile failed for rule '" + c.getName() + "' in " + sourceName,
t, Collections.emptyList());
@@ -172,12 +199,51 @@ public class LalFileApplier {
factory.addOrReplace(x);
registered.add(new RegisteredRule(x.layer, x.ruleName));
} catch (final Throwable t) {
+ // Roll back registrations made so far AND the layer claims. Rule-registration
+ // partial state survives in the caller's `partial` list for unwinding.
+ layerRegistry.rollback(appliedClaims);
throw new ApplyException(
"LAL register failed for rule '" + x.ruleName + "' in " + sourceName,
t, Collections.unmodifiableList(new ArrayList<>(registered)));
}
}
- return new Applied(sourceName, Collections.unmodifiableList(registered), ruleLoader);
+ return new Applied(sourceName, Collections.unmodifiableList(registered), ruleLoader,
+ appliedClaims);
+ }
+
+ /** Split {@code "catalog/name"} → catalog half. Falls back to {@code lal} for bare
+ * source names (legacy / test callers). */
+ private static String deriveCatalog(final String sourceName) {
+ final int idx = sourceName.indexOf('/');
+ return idx > 0 ? sourceName.substring(0, idx) : "lal";
+ }
+
+ private static String deriveRuleName(final String sourceName) {
+ final int idx = sourceName.indexOf('/');
+ return idx > 0 ? sourceName.substring(idx + 1) : sourceName;
+ }
+
+ /**
+ * Build {@link LayerClaim}s for every entry in {@code configs.layerDefinitions}.
+ * Empty input returns an empty list — callers may pass it straight to
+ * {@link RuntimeLayerRegistry#apply}, which is a no-op for an empty list.
+ *
+ * <p>{@code ordinal:} is mandatory in runtime DSL — the registry-side validation
+ * rejects any entry whose ordinal is below {@link Layer#RUNTIME_DYNAMIC_MIN_ORDINAL}
+ * (including the default {@code 0} that an omitted-yaml-key produces) with applyStatus
+ * {@code layer_ordinal_out_of_range}. {@code normal:} is optional and defaults to
+ * {@code true}.
+ */
+ private static List<LayerClaim> extractLayerClaims(final LALConfigs configs) {
+ final List<LayerDefinition> defs = configs.getLayerDefinitions();
+ if (defs == null || defs.isEmpty()) {
+ return Collections.emptyList();
+ }
+ final List<LayerClaim> out = new ArrayList<>(defs.size());
+ for (final LayerDefinition def : defs) {
+ out.add(new LayerClaim(def.getName(), def.getOrdinal(), def.isNormal()));
+ }
+ return out;
}
/**
@@ -249,7 +315,7 @@ public class LalFileApplier {
}
}
- private List<LALConfig> parse(final String yamlContent, final String sourceName) throws ApplyException {
+ private LALConfigs parse(final String yamlContent, final String sourceName) throws ApplyException {
try (StringReader reader = new StringReader(yamlContent)) {
final LALConfigs configs = new Yaml().loadAs(reader, LALConfigs.class);
if (configs == null || configs.getRules() == null || configs.getRules().isEmpty()) {
@@ -257,16 +323,10 @@ public class LalFileApplier {
"LAL YAML parsed to empty/malformed — no rules list in " + sourceName,
null, Collections.emptyList());
}
- if (configs.getLayerDefinitions() != null && !configs.getLayerDefinitions().isEmpty()) {
- throw new ApplyException(
- "LAL rule " + sourceName + " declares `layerDefinitions:`, which is only "
- + "permitted in bundled rule files loaded at OAP boot. Layer extensions "
- + "cannot be added at runtime — the registry is sealed once OAP startup "
- + "completes. Declare the layer in `layer-extensions.yml` or a bundled "
- + "MAL/LAL file, restart OAP, then retry this update.",
- null, Collections.emptyList());
- }
- return configs.getRules();
+ // layerDefinitions: are now permitted in runtime LAL rules; the apply path
+ // funnels them through the runtime-layer registry. The rejection that used to
+ // live here was removed when runtime dynamic layers became a first-class feature.
+ return configs;
} catch (final ApplyException e) {
throw e;
} catch (final Throwable t) {
@@ -290,16 +350,27 @@ public class LalFileApplier {
*/
@Getter
private final RuleClassLoader ruleClassLoader;
+ /** {@code null} when this LAL file declared no {@code layerDefinitions:}; otherwise
+ * the rollback token for the runtime-layer registry mutation this apply effected. */
+ @Getter
+ private final AppliedClaims appliedLayerClaims;
public Applied(final String sourceName, final List<RegisteredRule> registered) {
- this(sourceName, registered, null);
+ this(sourceName, registered, null, null);
}
public Applied(final String sourceName, final List<RegisteredRule> registered,
final RuleClassLoader ruleClassLoader) {
+ this(sourceName, registered, ruleClassLoader, null);
+ }
+
+ public Applied(final String sourceName, final List<RegisteredRule> registered,
+ final RuleClassLoader ruleClassLoader,
+ final AppliedClaims appliedLayerClaims) {
this.sourceName = sourceName;
this.registered = registered;
this.ruleClassLoader = ruleClassLoader;
+ this.appliedLayerClaims = appliedLayerClaims;
}
@Override
@@ -365,6 +436,11 @@ public class LalFileApplier {
return Collections.emptySet();
}
+ @Override
+ public AppliedClaims appliedLayerClaims() {
+ return appliedLayerClaims;
+ }
+
private List<String> ruleKeys() {
final List<String> keys = new ArrayList<>(registered.size());
for (final RegisteredRule r : registered) {
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/MalFileApplier.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/MalFileApplier.java
index fb015f6f45..25d1c349a7 100644
--- a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/MalFileApplier.java
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/MalFileApplier.java
@@ -19,9 +19,11 @@
package org.apache.skywalking.oap.server.receiver.runtimerule.apply;
import java.io.StringReader;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
+import java.util.List;
import java.util.Map;
import java.util.Set;
import javassist.ClassPool;
@@ -32,12 +34,16 @@ import org.apache.skywalking.oap.meter.analyzer.v2.MetricConvert;
import org.apache.skywalking.oap.meter.analyzer.v2.prometheus.rule.MetricsRule;
import org.apache.skywalking.oap.meter.analyzer.v2.prometheus.rule.Rule;
import org.apache.skywalking.oap.server.core.CoreModule;
+import org.apache.skywalking.oap.server.core.analysis.LayerDefinition;
import org.apache.skywalking.oap.server.core.analysis.meter.MeterSystem;
import org.apache.skywalking.oap.server.core.classloader.Catalog;
import org.apache.skywalking.oap.server.core.classloader.DSLClassLoaderManager;
import org.apache.skywalking.oap.server.core.classloader.RuleClassLoader;
import org.apache.skywalking.oap.server.core.storage.model.StorageManipulationOpt;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.AppliedClaims;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.LayerClaim;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.RuntimeLayerRegistry;
import org.apache.skywalking.oap.server.receiver.runtimerule.state.EngineApplied;
import org.yaml.snakeyaml.Yaml;
@@ -63,9 +69,18 @@ import org.yaml.snakeyaml.Yaml;
public class MalFileApplier {
private final MeterSystem meterSystem;
+ private final RuntimeLayerRegistry layerRegistry;
public MalFileApplier(final MeterSystem meterSystem) {
+ this(meterSystem, RuntimeLayerRegistry.INSTANCE);
+ }
+
+ /** Test-friendly constructor: injects a dedicated layer registry so unit / integration
+ * tests don't share JVM-wide state with sibling tests. Production callers go through
+ * the singleton-binding constructor above. */
+ public MalFileApplier(final MeterSystem meterSystem, final RuntimeLayerRegistry layerRegistry) {
this.meterSystem = meterSystem;
+ this.layerRegistry = layerRegistry;
}
/**
@@ -100,6 +115,18 @@ public class MalFileApplier {
final DSLClassLoaderManager.Kind kind) throws ApplyException {
final Rule rule = parse(yamlContent, sourceName);
final Set<String> metricNames = enumerateMetricNames(rule);
+ final List<LayerClaim> layerClaims = extractLayerClaims(rule);
+
+ // Register any declared layers FIRST. Conflicts throw LayerConflictException
+ // (caller translates to HTTP 400 with the structured envelope); per-apply atomic
+ // because validate() runs before any registration lands. After this call, the
+ // Layer registry knows the new layer names; MeterSystem registration below can
+ // safely reference them. On any failure further down we roll back through
+ // RuntimeLayerRegistry.rollback(appliedClaims).
+ final AppliedClaims appliedClaims =
+ layerRegistry.apply(RuntimeLayerRegistry.ruleId(deriveCatalog(sourceName),
+ deriveRuleName(sourceName)),
+ layerClaims);
// Per-file RuleClassLoader + Javassist ClassPool. The pool is parented to the default
// pool so shipped ancestor classes (SumFunction, HistogramFunction, ...) resolve via
@@ -127,7 +154,9 @@ public class MalFileApplier {
// MeterSystem — the caller uses this set for rollback. Passing the full enumerated
// set here would remove metrics the old bundle still owns (disastrous on
// FILTER_ONLY edits, where by definition every metric name is also in the old
- // bundle).
+ // bundle). The layer-registry changes are reverted now so a failed MeterSystem
+ // register does not leak a half-applied layer state.
+ layerRegistry.rollback(appliedClaims);
throw new ApplyException(
"MAL register failed for " + sourceName + " (partial)",
pre.getCause() == null ? pre : pre.getCause(),
@@ -136,9 +165,22 @@ public class MalFileApplier {
// Phase-1 compile failure or other pre-register throw. Nothing was registered with
// MeterSystem, so rollback set is empty — passing a non-empty set would cause the
// caller to unregister metrics the old bundle owns and this apply never touched.
+ layerRegistry.rollback(appliedClaims);
throw new ApplyException("MAL compile failed for " + sourceName, t, Collections.emptySet());
}
- return new Applied(rule, convert, metricNames, ruleLoader);
+ return new Applied(rule, convert, metricNames, ruleLoader, appliedClaims);
+ }
+
+ /** Split {@code "catalog/name"} → catalog half. Falls back to {@code otel-rules} when the
+ * source name is bare (legacy callers, tests). */
+ private static String deriveCatalog(final String sourceName) {
+ final int idx = sourceName.indexOf('/');
+ return idx > 0 ? sourceName.substring(0, idx) : "otel-rules";
+ }
+
+ private static String deriveRuleName(final String sourceName) {
+ final int idx = sourceName.indexOf('/');
+ return idx > 0 ? sourceName.substring(idx + 1) : sourceName;
}
/**
@@ -217,15 +259,9 @@ public class MalFileApplier {
if (rule.getName() == null || rule.getName().isEmpty()) {
rule.setName(sourceName);
}
- if (rule.getLayerDefinitions() != null && !rule.getLayerDefinitions().isEmpty()) {
- throw new ApplyException(
- "MAL rule " + sourceName + " declares `layerDefinitions:`, which is only "
- + "permitted in bundled rule files loaded at OAP boot. Layer extensions "
- + "cannot be added at runtime — the registry is sealed once OAP startup "
- + "completes. Declare the layer in `layer-extensions.yml` or a bundled "
- + "MAL/LAL file, restart OAP, then retry this update.",
- null, Collections.emptySet());
- }
+ // layerDefinitions: are now permitted in runtime MAL rules; they're handled by
+ // the layer registry on the apply path below. The rejection that used to live
+ // here was removed when runtime dynamic layers became a first-class feature.
return rule;
} catch (final ApplyException e) {
throw e;
@@ -234,6 +270,32 @@ public class MalFileApplier {
}
}
+ /**
+ * Build {@link LayerClaim}s for every {@code layerDefinitions:} entry in {@code rule}.
+ * The list is ordered the same as the YAML for stable diagnostic output.
+ *
+ * <p>{@code ordinal:} is mandatory in runtime DSL — the registry-side validation rejects
+ * any entry whose ordinal is below {@link Layer#RUNTIME_DYNAMIC_MIN_ORDINAL} (including
+ * the default {@code 0} that an omitted-yaml-key produces) with applyStatus
+ * {@code layer_ordinal_out_of_range}. {@code normal:} is optional and defaults to
+ * {@code true} via {@link LayerDefinition}'s field initializer — same semantics as the
+ * bundled {@code layerDefinitions:} blocks and {@code layer-extensions.yml}.
+ *
+ * <p>An empty {@code layerDefinitions:} block returns an empty list — callers may pass
+ * it straight to {@link RuntimeLayerRegistry#apply}, which is a no-op for empty input.
+ */
+ private static List<LayerClaim> extractLayerClaims(final Rule rule) {
+ final List<LayerDefinition> defs = rule.getLayerDefinitions();
+ if (defs == null || defs.isEmpty()) {
+ return Collections.emptyList();
+ }
+ final List<LayerClaim> out = new ArrayList<>(defs.size());
+ for (final LayerDefinition def : defs) {
+ out.add(new LayerClaim(def.getName(), def.getOrdinal(), def.isNormal()));
+ }
+ return out;
+ }
+
/**
* Parse raw YAML and return the set of metric names the rule file would register.
* Static-only variant — does not compile, does not register, does not construct a
@@ -299,14 +361,33 @@ public class MalFileApplier {
*/
@Getter
private final RuleClassLoader ruleClassLoader;
+ /**
+ * Snapshot of the {@link RuntimeLayerRegistry} mutation this apply effected. {@code
+ * null} for synthetic {@link Applied}s built by {@code recordBundledClaims} (boot-
+ * time static-rule shadow), which never touch runtime layers. The orchestrator's
+ * commit / discard tail calls {@link RuntimeLayerRegistry#rollback} with this
+ * token on external failure (persist, peer-rejected commit); inside
+ * {@link MalFileApplier#apply} a thrown {@code MetricConvert} construction already
+ * rolls back before re-throwing.
+ */
+ @Getter
+ private final AppliedClaims appliedLayerClaims;
public Applied(final Rule rule, final MetricConvert metricConvert,
final Set<String> registeredMetricNames,
final RuleClassLoader ruleClassLoader) {
+ this(rule, metricConvert, registeredMetricNames, ruleClassLoader, null);
+ }
+
+ public Applied(final Rule rule, final MetricConvert metricConvert,
+ final Set<String> registeredMetricNames,
+ final RuleClassLoader ruleClassLoader,
+ final AppliedClaims appliedLayerClaims) {
this.rule = rule;
this.metricConvert = metricConvert;
this.registeredMetricNames = registeredMetricNames;
this.ruleClassLoader = ruleClassLoader;
+ this.appliedLayerClaims = appliedLayerClaims;
}
@Override
@@ -362,6 +443,11 @@ public class MalFileApplier {
? Collections.emptySet()
: registeredMetricNames;
}
+
+ @Override
+ public AppliedClaims appliedLayerClaims() {
+ return appliedLayerClaims;
+ }
}
/**
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/engine/lal/LalRuleEngine.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/engine/lal/LalRuleEngine.java
index 54ebcc61c2..4bb08ea1ac 100644
--- a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/engine/lal/LalRuleEngine.java
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/engine/lal/LalRuleEngine.java
@@ -43,6 +43,8 @@ import org.apache.skywalking.oap.server.receiver.runtimerule.state.EngineApplied
import org.apache.skywalking.oap.server.library.module.ModuleManager;
import org.apache.skywalking.oap.server.receiver.runtimerule.apply.DeltaClassifier;
import org.apache.skywalking.oap.server.receiver.runtimerule.apply.LalFileApplier;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.AppliedClaims;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.RuntimeLayerRegistry;
import org.apache.skywalking.oap.server.receiver.runtimerule.engine.ApplyInputs;
import org.apache.skywalking.oap.server.receiver.runtimerule.engine.Classification;
import org.apache.skywalking.oap.server.receiver.runtimerule.engine.CompiledDSL;
@@ -413,6 +415,14 @@ public final class LalRuleEngine implements RuleEngine<LalApplyContext> {
@Override
public void rollback(final CompiledDSL compiled, final LalApplyContext ctx) {
final CompiledLalDSL c = (CompiledLalDSL) compiled;
+ // Drop layer claims first so a failed factory rollback below does not strand
+ // runtime-layer state. Safe ordering: layer registry mutations are independent
+ // of the LogFilter factory's rule-key map.
+ final AppliedClaims layerClaims = c.getNewApplied() == null
+ ? null : c.getNewApplied().appliedLayerClaims();
+ if (layerClaims != null && !layerClaims.isNoOp()) {
+ RuntimeLayerRegistry.INSTANCE.rollback(layerClaims);
+ }
if (c.getNewApplied() == null || c.getNewApplied().getRegistered().isEmpty()) {
return;
}
@@ -490,6 +500,10 @@ public final class LalRuleEngine implements RuleEngine<LalApplyContext> {
public void unregister(final String catalog, final String name, final LalApplyContext ctx) {
final String key = DSLScriptKey.key(catalog, name);
final String sourceName = catalog + "/" + name;
+ // Drop every layer this rule claimed. Net-zero layers cascade to
+ // Layer.unregisterDynamic. Independent of the factory teardown below.
+ RuntimeLayerRegistry.INSTANCE.removeRule(
+ RuntimeLayerRegistry.ruleId(catalog, name));
final LalFileApplier.Applied priorLal = appliedFor(ctx.getRules(), key);
if (priorLal != null) {
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/engine/mal/MalRuleEngine.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/engine/mal/MalRuleEngine.java
index 0d6032db66..7da4c32dbb 100644
--- a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/engine/mal/MalRuleEngine.java
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/engine/mal/MalRuleEngine.java
@@ -51,6 +51,8 @@ import org.apache.skywalking.oap.server.core.storage.model.StorageManipulationOp
import org.apache.skywalking.oap.server.receiver.runtimerule.apply.DSLDelta;
import org.apache.skywalking.oap.server.receiver.runtimerule.apply.DeltaClassifier;
import org.apache.skywalking.oap.server.receiver.runtimerule.apply.MalFileApplier;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.AppliedClaims;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.RuntimeLayerRegistry;
import org.apache.skywalking.oap.server.receiver.runtimerule.engine.ApplyInputs;
import org.apache.skywalking.oap.server.receiver.runtimerule.engine.Classification;
import org.apache.skywalking.oap.server.receiver.runtimerule.engine.CompiledDSL;
@@ -606,6 +608,14 @@ public final class MalRuleEngine implements RuleEngine<MalApplyContext> {
@Override
public void rollback(final CompiledDSL compiled, final MalApplyContext ctx) {
final CompiledMalDSL c = (CompiledMalDSL) compiled;
+ // Drop layer claims FIRST so a failed registry rollback doesn't strand metrics
+ // pointing at layers we don't know about. The registry's rollback is bounded —
+ // worst case it logs and proceeds — so this ordering is safe.
+ final AppliedClaims layerClaims = c.getNewApplied() == null
+ ? null : c.getNewApplied().appliedLayerClaims();
+ if (layerClaims != null && !layerClaims.isNoOp()) {
+ RuntimeLayerRegistry.INSTANCE.rollback(layerClaims);
+ }
if (c.getAddedPlusShapeBreak().isEmpty()) {
return;
}
@@ -658,6 +668,11 @@ public final class MalRuleEngine implements RuleEngine<MalApplyContext> {
public void unregister(final String catalog, final String name, final MalApplyContext ctx) {
final String key = DSLScriptKey.key(catalog, name);
final String sourceName = catalog + "/" + name;
+ // Drop every layer this rule claimed. Net-zero layers cascade through
+ // RuntimeLayerRegistry.removeRule → Layer.unregisterDynamic. Safe to call before
+ // the metric teardown below — layers and metrics are independent registries.
+ RuntimeLayerRegistry.INSTANCE.removeRule(
+ RuntimeLayerRegistry.ruleId(catalog, name));
// Always drop the MalConverterRegistry entry for this (catalog, name). The key
// namespace is shared between boot-time and runtime converters, so this single call
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/extension/DbOverrideRuntimeRuleResolver.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/extension/DbOverrideRuntimeRuleResolver.java
index 0e5eb6b3e5..a8e4645c61 100644
--- a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/extension/DbOverrideRuntimeRuleResolver.java
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/extension/DbOverrideRuntimeRuleResolver.java
@@ -19,16 +19,22 @@
package org.apache.skywalking.oap.server.receiver.runtimerule.extension;
import java.io.IOException;
+import java.io.StringReader;
import java.nio.charset.StandardCharsets;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.log.analyzer.v2.provider.LALConfigs;
+import org.apache.skywalking.oap.meter.analyzer.v2.prometheus.rule.Rule;
+import org.apache.skywalking.oap.server.core.analysis.LayerDefinition;
import org.apache.skywalking.oap.server.core.rule.ext.RuntimeRuleOverrideResolver;
+import org.apache.skywalking.oap.server.core.rule.ext.StaticRuleRegistry;
import org.apache.skywalking.oap.server.core.storage.StorageModule;
import org.apache.skywalking.oap.server.core.storage.management.RuntimeRuleManagementDAO;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.yaml.snakeyaml.Yaml;
/**
* Serves operator-supplied rule overrides (rows in the {@code runtime_rule} management table)
@@ -145,14 +151,81 @@ public final class DbOverrideRuntimeRuleResolver implements RuntimeRuleOverrideR
if ("INACTIVE".equalsIgnoreCase(row.getStatus())) {
result.put(row.getName(), Resolution.inactive());
} else {
- // STATUS_ACTIVE (default) or any non-INACTIVE status — treat as active substitution.
+ // STATUS_ACTIVE (default) or any non-INACTIVE status — active substitution.
final byte[] bytes = row.getContent() == null
? new byte[0]
: row.getContent().getBytes(StandardCharsets.UTF_8);
+ // Layer overrides are never permitted on top of a bundled rule. If the
+ // override carries layerDefinitions AND the disk twin exists, drop the
+ // override so the bundled content loads as-is. Operators get the boot
+ // error; the /addOrUpdate REST handler rejects the same combination
+ // pre-persist for fresh pushes.
+ if (isLayerOverrideOnBundled(catalog, row.getName(), bytes)) {
+ log.error("runtime-rule boot resolver: rule {}/{} declares layerDefinitions "
+ + "AND has a bundled disk twin. The substitution is dropped so the "
+ + "static loader serves the bundled disk content (bundled layer "
+ + "ownership preserved). The runtime row is still in the DAO and will "
+ + "be applied dynamically post-seal by RuleSync (its layerDefinitions "
+ + "go through the runtime channel and are operator-removable via "
+ + "/inactivate + /delete). Operator action: remove layerDefinitions "
+ + "from the runtime row, OR /delete the row and re-create it as a "
+ + "pure-runtime rule with a different name.",
+ catalog, row.getName());
+ continue;
+ }
result.put(row.getName(), Resolution.active(bytes));
}
}
log.info("Runtime-rule boot resolver loaded {} override(s) for catalog {}", result.size(), catalog);
return result;
}
+
+ /** True when content has a non-empty layerDefinitions block AND the disk twin
+ * exists. The static loader records every disk entry into {@link StaticRuleRegistry}
+ * before resolvers run, so the registry is authoritative for disk-twin presence. */
+ public static boolean isLayerOverrideOnBundled(final String catalog, final String name,
+ final byte[] bytes) {
+ if (bytes == null || bytes.length == 0) {
+ return false;
+ }
+ final StaticRuleRegistry registry = StaticRuleRegistry.active();
+ if (registry == null || !registry.find(catalog, name).isPresent()) {
+ return false;
+ }
+ return contentHasLayerDefinitions(catalog, new String(bytes, StandardCharsets.UTF_8));
+ }
+
+ /** YAML peek for the {@code layerDefinitions:} block. MAL catalogs bind into
+ * {@link Rule}; LAL into {@link LALConfigs}. Unknown catalogs return false (no
+ * catalog-specific parsing → can't enforce; caller treats as no layers). */
+ public static boolean contentHasLayerDefinitions(final String catalog, final String content) {
+ if (content == null || content.isEmpty()) {
+ return false;
+ }
+ try (StringReader reader = new StringReader(content)) {
+ final List<LayerDefinition> defs;
+ switch (catalog) {
+ case "otel-rules":
+ case "log-mal-rules":
+ case "telegraf-rules": {
+ final Rule rule = new Yaml().loadAs(reader, Rule.class);
+ defs = rule == null ? null : rule.getLayerDefinitions();
+ break;
+ }
+ case "lal": {
+ final LALConfigs configs = new Yaml().loadAs(reader, LALConfigs.class);
+ defs = configs == null ? null : configs.getLayerDefinitions();
+ break;
+ }
+ default:
+ return false;
+ }
+ return defs != null && !defs.isEmpty();
+ } catch (final RuntimeException ex) {
+ // YAML malformed: not our problem here. The applier surfaces the parse error
+ // with a clearer message; the layer-override check is a no-op for unparseable
+ // content (the apply will reject the row anyway).
+ return false;
+ }
+ }
}
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/AppliedClaims.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/AppliedClaims.java
new file mode 100644
index 0000000000..d6b2162553
--- /dev/null
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/AppliedClaims.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.receiver.runtimerule.layer;
+
+import java.util.Collections;
+import java.util.Set;
+import lombok.Getter;
+
+/**
+ * Snapshot of what a {@link RuntimeLayerRegistry#apply} call did, sufficient for
+ * {@link RuntimeLayerRegistry#rollback} to undo the change. Returned to the applier so it
+ * can hand the bundle to the orchestrator alongside the {@code Applied} artifact — the
+ * structural commit coordinator calls back into the registry with this token on persist /
+ * commit-tail failure.
+ *
+ * <p>{@link #getNewlyRegistered} is informational (which layers were brand-new to the
+ * registry, not just newly-claimed by this rule). Rollback uses the {@code priorClaims}
+ * and {@code currentLayerNames} diff to restore the exact state without needing the
+ * net-new set.
+ */
+@Getter
+public final class AppliedClaims {
+
+ private final String ruleId;
+ private final Set<LayerClaim> priorClaims;
+ private final Set<String> currentLayerNames;
+ private final Set<String> newlyRegistered;
+
+ AppliedClaims(final String ruleId,
+ final Set<LayerClaim> priorClaims,
+ final Set<String> currentLayerNames,
+ final Set<String> newlyRegistered) {
+ this.ruleId = ruleId;
+ this.priorClaims = priorClaims == null
+ ? Collections.emptySet()
+ : Collections.unmodifiableSet(priorClaims);
+ this.currentLayerNames = currentLayerNames == null
+ ? Collections.emptySet()
+ : Collections.unmodifiableSet(currentLayerNames);
+ this.newlyRegistered = newlyRegistered == null
+ ? Collections.emptySet()
+ : Collections.unmodifiableSet(newlyRegistered);
+ }
+
+ /** Convenience: true when the apply effected no actual change (empty old and new). */
+ public boolean isNoOp() {
+ return priorClaims.isEmpty() && currentLayerNames.isEmpty();
+ }
+}
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/LayerClaim.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/LayerClaim.java
new file mode 100644
index 0000000000..8b480d93d6
--- /dev/null
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/LayerClaim.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.receiver.runtimerule.layer;
+
+import java.util.Objects;
+import lombok.Getter;
+
+/**
+ * Immutable resolved layer declaration used inside the runtime-rule module. Produced from
+ * a MAL/LAL {@code layerDefinitions:} entry after the applier has resolved
+ * server-allocated ordinals (for entries with no explicit {@code ordinal:}) and validated
+ * the {@code >= RUNTIME_DYNAMIC_MIN_ORDINAL} floor.
+ *
+ * <p>Equality is by the full {@code (name, ordinal, normal)} triple — two claims are
+ * "same" iff all three fields match. This is the same notion of identity that
+ * {@link org.apache.skywalking.oap.server.core.analysis.Layer#registerDynamic} uses to
+ * decide whether re-registration is idempotent.
+ */
+@Getter
+public final class LayerClaim {
+
+ private final String name;
+ private final int ordinal;
+ private final boolean normal;
+
+ public LayerClaim(final String name, final int ordinal, final boolean normal) {
+ if (name == null) {
+ throw new IllegalArgumentException("layer name must not be null");
+ }
+ this.name = name;
+ this.ordinal = ordinal;
+ this.normal = normal;
+ }
+
+ @Override
+ public boolean equals(final Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof LayerClaim)) {
+ return false;
+ }
+ final LayerClaim that = (LayerClaim) o;
+ return ordinal == that.ordinal
+ && normal == that.normal
+ && name.equals(that.name);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(name, ordinal, normal);
+ }
+
+ @Override
+ public String toString() {
+ return "LayerClaim{name=" + name + ", ordinal=" + ordinal + ", normal=" + normal + "}";
+ }
+}
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/LayerConflictException.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/LayerConflictException.java
new file mode 100644
index 0000000000..017dcbf091
--- /dev/null
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/LayerConflictException.java
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.receiver.runtimerule.layer;
+
+import lombok.Getter;
+
+/**
+ * Typed exception raised by {@link RuntimeLayerConflictChecker} when a runtime DSL
+ * {@code layerDefinitions:} entry cannot be accepted. Carries a stable {@code applyStatus}
+ * code so the runtime-rule REST handler can map directly into the existing
+ * {@code {applyStatus, catalog, name, message}} response envelope without case analysis.
+ *
+ * <p>The {@code message} surfaces the offending value verbatim (layer name, conflicting
+ * ordinal, declaring source) so the operator's next action is obvious from the response
+ * body alone.
+ */
+@Getter
+public final class LayerConflictException extends RuntimeException {
+
+ /**
+ * Stable identifier mirrored as the {@code applyStatus} field on the HTTP response.
+ * Operators and automation should pattern-match on the enum constant name (which is
+ * exactly the wire value via {@link #wireValue()}).
+ */
+ public enum Status {
+ LAYER_ORDINAL_OUT_OF_RANGE("layer_ordinal_out_of_range"),
+ LAYER_NAME_CONFLICT("layer_name_conflict"),
+ LAYER_ORDINAL_COLLISION("layer_ordinal_collision"),
+ LAYER_NAME_INVALID("layer_name_invalid"),
+ /** Runtime override of a bundled rule attempted to declare layerDefinitions.
+ * Layer ownership lives with whichever channel registered it first; runtime
+ * overrides may change the rule body but must not touch layer declarations. */
+ LAYER_OVERRIDE_FORBIDDEN("layer_override_forbidden");
+
+ private final String wire;
+
+ Status(final String wire) {
+ this.wire = wire;
+ }
+
+ public String wireValue() {
+ return wire;
+ }
+ }
+
+ private final Status status;
+
+ public LayerConflictException(final Status status, final String message) {
+ super(message);
+ this.status = status;
+ }
+
+ /** Convenience accessor for callers building the HTTP response envelope. */
+ public String applyStatus() {
+ return status.wireValue();
+ }
+}
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/RuntimeLayerConflictChecker.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/RuntimeLayerConflictChecker.java
new file mode 100644
index 0000000000..65fa22a386
--- /dev/null
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/RuntimeLayerConflictChecker.java
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.receiver.runtimerule.layer;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import org.apache.skywalking.oap.server.core.analysis.Layer;
+
+/**
+ * Pure validation helper for runtime-DSL layer declarations. Produces operator-actionable
+ * {@link LayerConflictException}s that the REST handler translates 1:1 into the
+ * {@code {applyStatus, message}} response envelope. Every message names the offending
+ * layer, the conflicting ordinal, and (where applicable) the source that already owns the
+ * name, so the operator's corrective action is obvious from the response body alone.
+ *
+ * <p>Validation is performed against the live {@link Layer} registry, masking out the
+ * caller rule's prior claims so a same-rule edit (e.g. changing a runtime layer's ordinal
+ * from {@code 100_001} to {@code 100_002}) is permitted without an
+ * inactivate-then-recreate round trip.
+ */
+public final class RuntimeLayerConflictChecker {
+
+ /**
+ * Same regex enforced by {@link Layer#registerDynamic}; mirrored here so the operator
+ * gets the dedicated {@code layer_name_invalid} status (mapped to HTTP 400) instead
+ * of an {@code IllegalArgumentException} masquerading as a 500.
+ */
+ private static final Pattern NAME_PATTERN = Pattern.compile("[A-Z][A-Z0-9_]*");
+
+ private RuntimeLayerConflictChecker() {
+ }
+
+ /**
+ * Validate {@code candidates} against the current {@link Layer} registry and the
+ * given {@link RuntimeLayerRegistry}, treating the caller rule's {@code priorClaims}
+ * as already-removed (because the in-flight apply will replace them). Throws on the
+ * first detected conflict; partial applies are impossible because the caller invokes
+ * this BEFORE any {@link Layer#registerDynamic} call lands.
+ *
+ * @param ruleId canonical rule identifier — surfaced in conflict messages so
+ * the operator can pinpoint the source side of a cross-rule
+ * conflict
+ * @param candidates proposed layer declarations (ordinal already validated to
+ * {@code >= RUNTIME_DYNAMIC_MIN_ORDINAL} by the applier; this
+ * method re-checks for defence in depth)
+ * @param priorClaims layer names this rule currently claims; conflicts against
+ * these are NOT raised, since the apply will release them
+ * @param refcounts refcount tracker used to label runtime-side conflict sources
+ * ("runtime rule otel-rules/foo also declares X")
+ * @throws LayerConflictException on the first conflict
+ */
+ public static void validate(final String ruleId,
+ final List<LayerClaim> candidates,
+ final Set<String> priorClaims,
+ final RuntimeLayerRegistry refcounts) {
+ if (candidates == null || candidates.isEmpty()) {
+ return;
+ }
+ // Pre-pass: detect duplicates within the same batch BEFORE any registry check.
+ // Two entries naming the same layer with different triples, or two entries pinning
+ // the same ordinal under different names, cannot both succeed even in isolation;
+ // raising early keeps apply() atomic — without this pre-pass, a per-claim loop
+ // would mutate the registry for entry N before throwing on entry N+1, leaving a
+ // half-applied state that the rollback path then has to unwind.
+ final Map<String, LayerClaim> byName = new HashMap<>(candidates.size());
+ final Map<Integer, LayerClaim> byOrdinal = new HashMap<>(candidates.size());
+ for (final LayerClaim claim : candidates) {
+ final LayerClaim sameName = byName.get(claim.getName());
+ if (sameName != null && !sameName.equals(claim)) {
+ throw new LayerConflictException(
+ LayerConflictException.Status.LAYER_NAME_CONFLICT,
+ "Batch in rule " + ruleId + " declares layer '" + claim.getName()
+ + "' twice with different triples (" + sameName + " vs " + claim
+ + "). A single batch must declare each layer at most once.");
+ }
+ final LayerClaim sameOrdinal = byOrdinal.get(claim.getOrdinal());
+ if (sameOrdinal != null && !sameOrdinal.getName().equals(claim.getName())) {
+ throw new LayerConflictException(
+ LayerConflictException.Status.LAYER_ORDINAL_COLLISION,
+ "Batch in rule " + ruleId + " pins ordinal " + claim.getOrdinal()
+ + " under two different names ('" + sameOrdinal.getName()
+ + "' and '" + claim.getName() + "'). Each ordinal must be unique "
+ + "within a single layerDefinitions block.");
+ }
+ byName.put(claim.getName(), claim);
+ byOrdinal.put(claim.getOrdinal(), claim);
+ }
+ for (final LayerClaim claim : candidates) {
+ validateOne(ruleId, claim, priorClaims, refcounts);
+ }
+ }
+
+ private static void validateOne(final String ruleId,
+ final LayerClaim claim,
+ final Set<String> priorClaims,
+ final RuntimeLayerRegistry refcounts) {
+ // 1. Name shape.
+ if (claim.getName() == null || !NAME_PATTERN.matcher(claim.getName()).matches()) {
+ throw new LayerConflictException(
+ LayerConflictException.Status.LAYER_NAME_INVALID,
+ "Layer name '" + claim.getName() + "' is invalid — must match "
+ + "[A-Z][A-Z0-9_]* (upper-snake-case, leading letter, no spaces).");
+ }
+
+ // 2. Ordinal range.
+ if (claim.getOrdinal() < Layer.RUNTIME_DYNAMIC_MIN_ORDINAL) {
+ throw new LayerConflictException(
+ LayerConflictException.Status.LAYER_ORDINAL_OUT_OF_RANGE,
+ "Layer '" + claim.getName() + "' declared with ordinal " + claim.getOrdinal()
+ + " in rule " + ruleId + ". Runtime DSL layers must pin an explicit "
+ + "ordinal in the runtime tier (>= " + Layer.RUNTIME_DYNAMIC_MIN_ORDINAL
+ + "). Reserved ranges: 0-9999 = built-in Layer constants, "
+ + "10_000-99_999 = layer-extensions.yml + bundled MAL/LAL "
+ + "layerDefinitions. Omitting 'ordinal:' (default 0) is rejected "
+ + "because the ordinal is persisted in ServiceTraffic primary keys "
+ + "and must be operator-stable across restarts.");
+ }
+
+ // 3. Name conflict. Non-dynamic existing + same triple = soft claim (refcount
+ // tracks the runtime rule but Layer stays bundled). Anything else against a
+ // non-dynamic owner is a hard conflict. Against a dynamic owner, self-claims
+ // are permitted; cross-rule and multi-claimant triple changes are not.
+ final Layer byName = Layer.nameOf(claim.getName());
+ if (byName != Layer.UNDEFINED) {
+ final boolean tripleMatches =
+ byName.value() == claim.getOrdinal() && byName.isNormal() == claim.isNormal();
+ if (!Layer.isDynamic(claim.getName())) {
+ if (tripleMatches) {
+ // Soft claim — apply refcount-adds without calling registerDynamic.
+ return;
+ }
+ throw new LayerConflictException(
+ LayerConflictException.Status.LAYER_NAME_CONFLICT,
+ "Layer '" + claim.getName() + "' is already registered by "
+ + sourceLabel(claim.getName(), refcounts)
+ + " as (ordinal=" + byName.value()
+ + ", normal=" + byName.isNormal()
+ + ") through the boot-time channel; rule " + ruleId
+ + " declares (ordinal=" + claim.getOrdinal()
+ + ", normal=" + claim.isNormal() + "). Align the runtime "
+ + "declaration to match, or rename the runtime layer.");
+ }
+ if (!tripleMatches) {
+ final boolean priorlyClaimedByThisRule =
+ priorClaims != null && priorClaims.contains(claim.getName());
+ if (priorlyClaimedByThisRule) {
+ // Self-update is only legal if this rule is the sole claimant.
+ final Set<String> claimants = refcounts.claimantsOf(claim.getName());
+ if (claimants.size() > 1) {
+ throw new LayerConflictException(
+ LayerConflictException.Status.LAYER_NAME_CONFLICT,
+ "Layer '" + claim.getName() + "' is also claimed by other "
+ + "runtime rule(s) "
+ + String.join(", ", otherClaimants(claimants, ruleId))
+ + " at (ordinal=" + byName.value()
+ + ", normal=" + byName.isNormal()
+ + "); rule " + ruleId + " cannot change the triple to "
+ + "(ordinal=" + claim.getOrdinal()
+ + ", normal=" + claim.isNormal()
+ + ") because the change would affect the other claimants. "
+ + "Remove the prior declaration from this rule first, or "
+ + "have every claimant align on the new triple in lockstep.");
+ }
+ // Sole claimant — apply will replace the prior triple. Fall through.
+ } else {
+ throw new LayerConflictException(
+ LayerConflictException.Status.LAYER_NAME_CONFLICT,
+ "Layer '" + claim.getName() + "' already registered by "
+ + sourceLabel(claim.getName(), refcounts)
+ + " as (ordinal=" + byName.value()
+ + ", normal=" + byName.isNormal()
+ + "); rule " + ruleId + " declares (ordinal=" + claim.getOrdinal()
+ + ", normal=" + claim.isNormal()
+ + "). Align both declarations or rename one.");
+ }
+ } else {
+ // Same name and same triple — idempotent re-declare (dynamic only,
+ // because the !isDynamic branch above already rejected the bundled case).
+ return;
+ }
+ }
+
+ // 4. Ordinal collision — different name, same ordinal already used by someone.
+ // Sub-cases mirror step 3: bundled holders are immovable; dynamic holders allow
+ // a self-update where this rule is freeing the ordinal.
+ final Layer byOrdinal = Layer.peekByValue(claim.getOrdinal());
+ if (byOrdinal != null && !byOrdinal.name().equals(claim.getName())) {
+ if (!Layer.isDynamic(byOrdinal.name())) {
+ throw new LayerConflictException(
+ LayerConflictException.Status.LAYER_ORDINAL_COLLISION,
+ "Layer '" + claim.getName() + "' ordinal " + claim.getOrdinal()
+ + " is already used by layer '" + byOrdinal.name()
+ + "' (registered by " + sourceLabel(byOrdinal.name(), refcounts)
+ + " through the boot-time channel). Pick a different ordinal "
+ + "(>= " + Layer.RUNTIME_DYNAMIC_MIN_ORDINAL + ").");
+ }
+ final boolean priorlyHeldByThisRule =
+ priorClaims != null && priorClaims.contains(byOrdinal.name());
+ if (!priorlyHeldByThisRule) {
+ throw new LayerConflictException(
+ LayerConflictException.Status.LAYER_ORDINAL_COLLISION,
+ "Layer '" + claim.getName() + "' ordinal " + claim.getOrdinal()
+ + " is already used by layer '" + byOrdinal.name()
+ + "' (registered by " + sourceLabel(byOrdinal.name(), refcounts)
+ + "). Pick a different ordinal (>= "
+ + Layer.RUNTIME_DYNAMIC_MIN_ORDINAL + ").");
+ }
+ // Self-update freeing the ordinal is only legal if this rule is the sole
+ // claimant of the prior layer. Other claimants would still hold the slot
+ // after apply Pass-1 drops THIS rule's claim, so registerDynamic would
+ // throw an unstructured collision deep inside the apply path.
+ final Set<String> otherHolders = refcounts.claimantsOf(byOrdinal.name());
+ if (otherHolders.size() > 1) {
+ throw new LayerConflictException(
+ LayerConflictException.Status.LAYER_ORDINAL_COLLISION,
+ "Layer '" + claim.getName() + "' ordinal " + claim.getOrdinal()
+ + " is still held by layer '" + byOrdinal.name()
+ + "' under other runtime rule(s) "
+ + String.join(", ", otherClaimants(otherHolders, ruleId))
+ + "; rule " + ruleId + " cannot reclaim the ordinal alone. "
+ + "Either drop the prior layer from this rule with the others "
+ + "in lockstep, or pick a different ordinal.");
+ }
+ }
+ }
+
+ /** Filter {@code claimants} to entries other than {@code self}. Used in self-update
+ * conflict messages to name only the rules whose declaration would be impacted. */
+ private static List<String> otherClaimants(final Set<String> claimants, final String self) {
+ final List<String> out = new ArrayList<>(claimants.size());
+ for (final String c : claimants) {
+ if (!c.equals(self)) {
+ out.add(c);
+ }
+ }
+ return out;
+ }
+
+ /**
+ * Label the source that owns a layer's current registration. Granularity follows the
+ * {@link Layer#isDynamic} channel, NOT the ordinal range — operator yaml may put a
+ * boot-time layer anywhere the registry accepts, including the {@code >=100_000}
+ * tier; labelling those as "runtime DSL" would mislead the operator into thinking
+ * they could realign at runtime.
+ * <ul>
+ * <li>{@code "built-in or boot-time extension"} — registered through
+ * {@link Layer#register} (built-in {@code Layer.*} constants,
+ * {@code layer-extensions.yml}, bundled MAL/LAL {@code layerDefinitions:}).</li>
+ * <li>{@code "runtime rule(s) <id1>, <id2>, ..."} — registered through
+ * {@link Layer#registerDynamic}, listing every runtime rule currently claiming
+ * the name through the refcount tracker.</li>
+ * </ul>
+ */
+ private static String sourceLabel(final String name, final RuntimeLayerRegistry refcounts) {
+ final Layer layer = Layer.nameOf(name);
+ if (layer == Layer.UNDEFINED) {
+ return "(unknown source — registry inconsistency)";
+ }
+ if (!Layer.isDynamic(name)) {
+ return "built-in or boot-time extension";
+ }
+ final Set<String> claimants = refcounts.claimantsOf(name);
+ if (claimants.isEmpty()) {
+ // Edge case: layer is in the runtime tier but not tracked by refcount. Should
+ // not happen under the synchronized boundary; log-and-fall-back if it does.
+ return "runtime DSL (untracked)";
+ }
+ return "runtime rule(s) " + String.join(", ", claimants);
+ }
+}
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/RuntimeLayerRegistry.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/RuntimeLayerRegistry.java
new file mode 100644
index 0000000000..ddb0d310f7
--- /dev/null
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/RuntimeLayerRegistry.java
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.receiver.runtimerule.layer;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.analysis.Layer;
+
+/**
+ * In-memory refcount tracker for runtime-DSL dynamic layer declarations. Sits between the
+ * runtime-rule appliers and {@link Layer}'s post-seal {@link Layer#registerDynamic} /
+ * {@link Layer#unregisterDynamic} entry points.
+ *
+ * <p>Lifecycle invariant: a layer is present in {@link Layer}'s registry (with ordinal
+ * {@code >= RUNTIME_DYNAMIC_MIN_ORDINAL}) iff at least one runtime rule currently claims
+ * it through this registry. When the last claim drops, the layer is unregistered. Layers
+ * with ordinals below {@link Layer#RUNTIME_DYNAMIC_MIN_ORDINAL} are never tracked here —
+ * they are owned by the bundled / boot-time channels and are non-removable.
+ *
+ * <p>Crash recovery: state lives in-memory only. On OAP restart the runtime-rule replay
+ * path re-applies every stored runtime rule through this registry, rebuilding the same
+ * refcount snapshot. No separate persistence is needed because the rule yaml itself is
+ * the source of truth for layer declarations.
+ *
+ * <p>Cluster semantics: each OAP instance owns an independent {@code RuntimeLayerRegistry}.
+ * Convergence comes from operator-pinned ordinals (same yaml on every node produces the
+ * same registration) plus rule yaml replicating identically through the existing
+ * runtime-rule storage. Refcount state is process-local — peers don't share the map; they
+ * each track their own claims.
+ *
+ * <p>Thread safety: every public method is {@code synchronized} so apply / rollback /
+ * teardown sequences are atomic. The registry is the single mutator of {@link Layer}'s
+ * dynamic entries inside the runtime-rule module — the appliers and orchestrator must not
+ * call {@link Layer#registerDynamic} directly.
+ */
+@Slf4j
+public final class RuntimeLayerRegistry {
+
+ /**
+ * Process-wide singleton mirroring {@link Layer}'s process-singleton scope. Used by the
+ * MAL/LAL engines from their compile / rollback / unregister paths. Tests that need
+ * an isolated registry can construct one via {@link #RuntimeLayerRegistry()}.
+ */
+ public static final RuntimeLayerRegistry INSTANCE = new RuntimeLayerRegistry();
+
+ /**
+ * Claim graph: layer name → set of rule IDs that have declared this layer. A layer's
+ * lifetime in {@link Layer}'s registry is bounded by this set being non-empty. Use
+ * {@link LinkedHashSet} so iteration order is stable for diagnostic output.
+ */
+ private final Map<String, Set<String>> claimsByLayer = new HashMap<>();
+
+ /**
+ * Reverse index: rule ID → set of layer names this rule currently claims. Maintained
+ * in lockstep with {@link #claimsByLayer} so a rule's prior claims can be looked up
+ * in {@code O(claims)} during UPDATE.
+ */
+ private final Map<String, Set<String>> claimsByRule = new HashMap<>();
+
+ /**
+ * Static helper for building the canonical rule ID consumed by every method on this
+ * registry. The MAL/LAL appliers receive {@code sourceName} as {@code "catalog/name"}
+ * already, so the simplest path is to pass that string through verbatim — keeping the
+ * registry decoupled from the {@code Catalog} enum.
+ */
+ public static String ruleId(final String catalog, final String name) {
+ return catalog + "/" + name;
+ }
+
+ /**
+ * Validate the proposed declarations against the current state, then apply them
+ * atomically: register net-new layers through {@link Layer#registerDynamic}, replace
+ * this rule's prior claims with {@code newClaims} in the refcount graph, and
+ * unregister any layer whose last claimant just dropped.
+ *
+ * <p>Atomicity is per-rule: if validation throws partway, no state changes are
+ * committed. If a {@link Layer#registerDynamic} call throws after some registrations
+ * have already landed (race against another OAP-internal mutator — should not happen
+ * under the synchronized boundary, defensive only), the partial state is rolled back
+ * before re-throwing.
+ *
+ * @param ruleId canonical rule identifier (use {@link #ruleId(String, String)})
+ * @param newClaims the resolved layer declarations from the rule yaml (post-allocation,
+ * post-validation of the {@code >= 100_000} ordinal floor)
+ * @return {@link AppliedClaims} carrying the prior state so the caller can roll back
+ * later (e.g. on persist / commit failure outside the applier).
+ * @throws LayerConflictException on any name/ordinal conflict against the current
+ * registry view (excluding this rule's own prior claims, so self-edits work).
+ */
+ public synchronized AppliedClaims apply(final String ruleId, final List<LayerClaim> newClaims) {
+ if (ruleId == null || ruleId.isEmpty()) {
+ throw new IllegalArgumentException("ruleId must be non-empty");
+ }
+ final Set<String> priorLayerNames = new LinkedHashSet<>(
+ claimsByRule.getOrDefault(ruleId, Collections.emptySet()));
+ // Snapshot BEFORE mutation — pass-1 removeClaim may unregister entries that
+ // priorLayerNamesSnapshot would no longer be able to resolve.
+ final Set<LayerClaim> priorClaimsSnapshot = priorLayerNamesSnapshot(priorLayerNames);
+
+ if (newClaims != null && !newClaims.isEmpty()) {
+ RuntimeLayerConflictChecker.validate(ruleId, newClaims, priorLayerNames, this);
+ }
+
+ // Diff plan computed up front so the two passes below are straight-line.
+ final Set<String> newLayerNames = new LinkedHashSet<>();
+ final Map<String, LayerClaim> newByName = new LinkedHashMap<>();
+ if (newClaims != null) {
+ for (final LayerClaim claim : newClaims) {
+ newLayerNames.add(claim.getName());
+ newByName.put(claim.getName(), claim);
+ }
+ }
+ final Set<String> toRemoveNames = new LinkedHashSet<>();
+ for (final LayerClaim prior : priorClaimsSnapshot) {
+ final LayerClaim incoming = newByName.get(prior.getName());
+ if (incoming == null || !incoming.equals(prior)) {
+ toRemoveNames.add(prior.getName());
+ }
+ }
+ final List<LayerClaim> toRegister = new ArrayList<>();
+ for (final LayerClaim claim : newByName.values()) {
+ final Layer existing = Layer.nameOf(claim.getName());
+ final boolean sameLiveTriple = existing != Layer.UNDEFINED
+ && existing.value() == claim.getOrdinal()
+ && existing.isNormal() == claim.isNormal();
+ if (sameLiveTriple) {
+ // Soft-claim case (existing is bundled) or dynamic-and-same-triple — no
+ // Layer mutation needed; refcount-add still happens in pass 2.
+ continue;
+ }
+ toRegister.add(claim);
+ }
+
+ // Pass 1: drop prior claims. Frees ordinals so swap-style replacements in pass 2
+ // can register against the now-free slot.
+ for (final String name : toRemoveNames) {
+ removeClaim(ruleId, name);
+ }
+
+ final Set<String> newlyRegisteredInThisApply = new LinkedHashSet<>();
+ try {
+ for (final LayerClaim claim : toRegister) {
+ final boolean wasAbsent = Layer.nameOf(claim.getName()) == Layer.UNDEFINED;
+ Layer.registerDynamic(claim.getName(), claim.getOrdinal(), claim.isNormal());
+ if (wasAbsent) {
+ newlyRegisteredInThisApply.add(claim.getName());
+ }
+ }
+ for (final LayerClaim claim : newClaims == null ? Collections.<LayerClaim>emptyList() : newClaims) {
+ claimsByLayer.computeIfAbsent(claim.getName(), k -> new LinkedHashSet<>())
+ .add(ruleId);
+ }
+ } catch (final RuntimeException e) {
+ // Defence-in-depth: synchronized + post-validate should make this unreachable.
+ for (final String name : newlyRegisteredInThisApply) {
+ try {
+ Layer.unregisterDynamic(name);
+ } catch (final RuntimeException ignore) {
+ }
+ }
+ for (final LayerClaim prior : priorClaimsSnapshot) {
+ try {
+ Layer.registerDynamic(prior.getName(), prior.getOrdinal(), prior.isNormal());
+ } catch (final RuntimeException ignore) {
+ }
+ claimsByLayer.computeIfAbsent(prior.getName(), k -> new LinkedHashSet<>())
+ .add(ruleId);
+ }
+ // Restore reverse index.
+ claimsByRule.put(ruleId, new LinkedHashSet<>(priorLayerNames));
+ throw e;
+ }
+
+ // Reverse index — final state matches new claims; intermediate Pass-1 mutations
+ // already cleaned out the prior names.
+ if (newLayerNames.isEmpty()) {
+ claimsByRule.remove(ruleId);
+ } else {
+ claimsByRule.put(ruleId, newLayerNames);
+ }
+
+ return new AppliedClaims(ruleId, priorClaimsSnapshot, newLayerNames,
+ newlyRegisteredInThisApply);
+ }
+
+ /**
+ * Undo the effect of an earlier {@link #apply(String, List)}: restore this rule's
+ * prior claims and unregister any layer that had been registered net-new by the apply
+ * and is no longer claimed.
+ *
+ * <p>Used by the orchestrator on persist / commit failure (after the applier had
+ * already succeeded). For in-applier failures the apply method's own internal
+ * rollback handles cleanup before re-throwing — callers do NOT need to invoke this
+ * for that case.
+ *
+ * @param applied bundle returned from the matching {@link #apply} call
+ */
+ public synchronized void rollback(final AppliedClaims applied) {
+ if (applied == null) {
+ return;
+ }
+ final String ruleId = applied.getRuleId();
+
+ // Drop the newly-claimed entries that the apply added.
+ for (final String name : applied.getCurrentLayerNames()) {
+ removeClaim(ruleId, name);
+ }
+
+ // Restore prior claims (re-registering layers if the apply had freed them).
+ for (final LayerClaim priorClaim : applied.getPriorClaims()) {
+ try {
+ Layer.registerDynamic(priorClaim.getName(), priorClaim.getOrdinal(),
+ priorClaim.isNormal());
+ } catch (final RuntimeException e) {
+ // Best-effort: the prior claim's layer might already be registered
+ // (idempotent re-register handles same-triple). Log and continue so the
+ // refcount graph stays correct even if one entry fails to re-register.
+ log.warn("runtime-layer rollback: failed to re-register prior claim {}",
+ priorClaim, e);
+ }
+ claimsByLayer.computeIfAbsent(priorClaim.getName(), k -> new LinkedHashSet<>())
+ .add(ruleId);
+ claimsByRule.computeIfAbsent(ruleId, k -> new LinkedHashSet<>())
+ .add(priorClaim.getName());
+ }
+
+ if (!claimsByRule.containsKey(ruleId)
+ || claimsByRule.get(ruleId).isEmpty()) {
+ claimsByRule.remove(ruleId);
+ }
+ }
+
+ /**
+ * Drop every claim owned by {@code ruleId}. Called by the unregister path on DELETE /
+ * INACTIVATE — the rule is gone, so its layers must release their refcount slot, and
+ * any layer whose last claimant just dropped is unregistered through
+ * {@link Layer#unregisterDynamic}.
+ */
+ public synchronized void removeRule(final String ruleId) {
+ final Set<String> owned = claimsByRule.remove(ruleId);
+ if (owned == null || owned.isEmpty()) {
+ return;
+ }
+ for (final String name : owned) {
+ final Set<String> claimants = claimsByLayer.get(name);
+ if (claimants == null) {
+ continue;
+ }
+ claimants.remove(ruleId);
+ if (claimants.isEmpty()) {
+ claimsByLayer.remove(name);
+ if (Layer.isDynamic(name)) {
+ try {
+ Layer.unregisterDynamic(name);
+ } catch (final RuntimeException e) {
+ log.warn("runtime-layer removeRule: unregister threw for {} "
+ + "(unexpected — Layer.isDynamic was true)", name, e);
+ }
+ }
+ // else: soft claim (bundled-tier owner). Refcount entry dropped; the
+ // bundled registration stays per the layer's lifecycle contract.
+ }
+ }
+ }
+
+ /**
+ * Snapshot of layer names this rule currently claims. Returns an empty unmodifiable
+ * set if the rule has no claims. Used by the conflict checker to exclude self-edits
+ * from conflict probes.
+ */
+ public synchronized Set<String> claimsOf(final String ruleId) {
+ final Set<String> owned = claimsByRule.get(ruleId);
+ if (owned == null || owned.isEmpty()) {
+ return Collections.emptySet();
+ }
+ return Collections.unmodifiableSet(new LinkedHashSet<>(owned));
+ }
+
+ /**
+ * Snapshot of all currently-claimed runtime dynamic layers. The map key is the layer
+ * name; the value is the set of rules that have declared it. Used by the conflict
+ * checker for source labelling ("runtime rule otel-rules/foo also declares this").
+ */
+ public synchronized Map<String, Set<String>> snapshot() {
+ final Map<String, Set<String>> copy = new LinkedHashMap<>();
+ for (final Map.Entry<String, Set<String>> e : claimsByLayer.entrySet()) {
+ copy.put(e.getKey(), Collections.unmodifiableSet(new LinkedHashSet<>(e.getValue())));
+ }
+ return Collections.unmodifiableMap(copy);
+ }
+
+ /** Lookup helper for the conflict checker: which rules claim this layer right now? */
+ synchronized Set<String> claimantsOf(final String layerName) {
+ final Set<String> claimants = claimsByLayer.get(layerName);
+ if (claimants == null) {
+ return Collections.emptySet();
+ }
+ return Collections.unmodifiableSet(new LinkedHashSet<>(claimants));
+ }
+
+ /** Remove {@code ruleId} from {@code layerName}'s claimants and from the reverse index;
+ * if the claimants set empties, delete the entry and unregister the layer IF the
+ * layer was registered through the dynamic channel. Soft claims (a runtime rule
+ * whose triple matches an existing bundled / boot-time layer — see the conflict
+ * checker's case 3a) do NOT unregister: the bundled tier owns the layer's lifecycle
+ * and only a restart-time removal of the boot-time source can drop it. */
+ private void removeClaim(final String ruleId, final String layerName) {
+ final Set<String> claimants = claimsByLayer.get(layerName);
+ if (claimants != null) {
+ claimants.remove(ruleId);
+ if (claimants.isEmpty()) {
+ claimsByLayer.remove(layerName);
+ if (Layer.isDynamic(layerName)) {
+ try {
+ Layer.unregisterDynamic(layerName);
+ } catch (final RuntimeException e) {
+ log.warn("runtime-layer apply: unregister threw for {} (unexpected — "
+ + "Layer.isDynamic was true)", layerName, e);
+ }
+ }
+ // else: non-dynamic layer (bundled / boot-time extension). Soft claim
+ // released; layer stays in the registry per the bundled-tier contract.
+ }
+ }
+ // Keep the reverse index in lockstep — without this, rollback re-adds prior
+ // claims on top of the (still-stale) currentLayerNames entries.
+ final Set<String> owned = claimsByRule.get(ruleId);
+ if (owned != null) {
+ owned.remove(layerName);
+ if (owned.isEmpty()) {
+ claimsByRule.remove(ruleId);
+ }
+ }
+ }
+
+ /**
+ * Re-resolve full prior claims (with ordinal/normal) from layer names. Used in
+ * {@link #apply} to build the prior-state component of {@link AppliedClaims} so
+ * {@link #rollback} can restore the exact triples. Walks {@link Layer}'s registry
+ * (lookups by name) because the registry is the live source of truth for
+ * {@code (ordinal, normal)}.
+ */
+ private Set<LayerClaim> priorLayerNamesSnapshot(final Set<String> priorLayerNames) {
+ if (priorLayerNames.isEmpty()) {
+ return Collections.emptySet();
+ }
+ final Set<LayerClaim> out = new LinkedHashSet<>();
+ for (final String name : priorLayerNames) {
+ final Layer existing = Layer.nameOf(name);
+ if (existing == Layer.UNDEFINED) {
+ // Shouldn't happen — we tracked this rule as claiming `name`, so it must
+ // have been registered. Skip rather than throw so rollback can proceed
+ // for the other entries.
+ log.warn("runtime-layer snapshot: prior-claimed layer {} not in registry "
+ + "(skipping during rollback bundle)", name);
+ continue;
+ }
+ out.add(new LayerClaim(existing.name(), existing.value(), existing.isNormal()));
+ }
+ return Collections.unmodifiableSet(out);
+ }
+}
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/reconcile/DSLRuntimeApply.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/reconcile/DSLRuntimeApply.java
index 31a0178287..3d924d37f6 100644
--- a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/reconcile/DSLRuntimeApply.java
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/reconcile/DSLRuntimeApply.java
@@ -165,6 +165,12 @@ public final class DSLRuntimeApply {
final CompiledDSL compiled;
try {
compiled = engine.compile(file, classification, kind, ctx);
+ } catch (final org.apache.skywalking.oap.server.receiver.runtimerule.layer.LayerConflictException lce) {
+ // Layer-declaration conflicts (operator-actionable, structured 400 at the REST
+ // layer) bypass the generic compile-failed path so RuntimeRuleService can
+ // surface the typed applyStatus + message. Re-thrown unchanged; the REST
+ // handler's catch already maps to badRequest with the structured envelope.
+ throw lce;
} catch (final EngineCompileException ece) {
log.error("runtime-rule apply: compile FAILED for {}/{}: {}",
file.getCatalog(), file.getName(), ece.getMessage(), ece);
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/rest/RuntimeRuleService.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/rest/RuntimeRuleService.java
index ff2f99475a..01d645a4e8 100644
--- a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/rest/RuntimeRuleService.java
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/rest/RuntimeRuleService.java
@@ -65,6 +65,8 @@ import org.apache.skywalking.oap.server.receiver.runtimerule.apply.DSLDelta;
import org.apache.skywalking.oap.server.receiver.runtimerule.apply.DeltaClassifier;
import org.apache.skywalking.oap.server.receiver.runtimerule.engine.Classification;
import org.apache.skywalking.oap.server.receiver.runtimerule.engine.RuleEngine;
+import org.apache.skywalking.oap.server.receiver.runtimerule.extension.DbOverrideRuntimeRuleResolver;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.LayerConflictException;
import org.apache.skywalking.oap.server.receiver.runtimerule.cluster.MainRouter;
import org.apache.skywalking.oap.server.receiver.runtimerule.cluster.RuntimeRuleClusterClient;
import org.apache.skywalking.oap.server.receiver.runtimerule.cluster.v1.ForwardResponse;
@@ -719,6 +721,19 @@ public class RuntimeRuleService {
if (content.isEmpty()) {
return badRequest("empty_body", catalog, name, "request body must be the raw rule content");
}
+ // Runtime overrides of bundled rules MUST NOT declare layerDefinitions. Layer
+ // ownership is single-source-of-truth — bundled rules own bundled-tier layers,
+ // runtime rules own runtime-tier layers, no overwrite either way. Reject pre-
+ // persist so the bad row never reaches the DAO.
+ if (DbOverrideRuntimeRuleResolver.isLayerOverrideOnBundled(
+ catalog, name, content.getBytes(StandardCharsets.UTF_8))) {
+ return badRequest("layer_override_forbidden", catalog, name,
+ "Runtime override of bundled rule " + catalog + "/" + name + " declared "
+ + "layerDefinitions. Bundled rules own their layer declarations; "
+ + "runtime overrides may only change the rule body. Remove the "
+ + "layerDefinitions block, or rename the rule so it becomes a pure-"
+ + "runtime rule (no bundled twin).");
+ }
// Single-main routing. Self is main → null → run local workflow. Non-main + not
// forwarded → forward to main and relay response. Non-main + forwarded → fail-safe
// 421 (cluster view split; refuse to re-forward).
@@ -900,6 +915,19 @@ public class RuntimeRuleService {
try {
postApply = dslManager.applyNowForRuleFile(ruleFile);
} catch (final Throwable t) {
+ // Layer conflict on FILTER_ONLY is rare — DeltaClassifier escalates
+ // layerDefinitions changes to STRUCTURAL precisely so this path doesn't see
+ // them post-persist. The defensive unwrap-and-surface guards against a
+ // classifier miss (e.g. yaml-key reorder that doesn't change the signature
+ // but bypasses the equality check); operators still see the structured 400.
+ final LayerConflictException lce = findLayerConflict(t);
+ if (lce != null) {
+ log.error("runtime-rule FILTER_ONLY apply hit a layer conflict POST-persist "
+ + "for {}/{} — the row is in DB but local apply was rejected. Operator "
+ + "must reconcile (DELETE + re-POST with aligned layerDefinitions). "
+ + "Conflict: {}", catalog, name, lce.getMessage());
+ return badRequest(lce.applyStatus(), catalog, name, lce.getMessage());
+ }
log.error("runtime-rule FILTER_ONLY apply failed after persist for {}/{} — DB "
+ "reflects the new content; this node will converge on the next dslManager "
+ "tick (same path peers use).", catalog, name, t);
@@ -980,7 +1008,27 @@ public class RuntimeRuleService {
final DSLRuntimeState postApply;
try {
postApply = dslManager.applyNowForRuleFile(ruleFile, true);
+ } catch (final LayerConflictException lce) {
+ // Runtime-DSL layer-declaration conflict — operator-actionable, not a server
+ // error. Resume local + peers immediately so the cluster keeps serving the
+ // prior bundle.
+ log.warn("runtime-rule STRUCTURAL apply rejected for {}/{} — layer conflict: {}",
+ catalog, name, lce.getMessage());
+ dslManager.getSuspendCoord().localResume(catalog, name);
+ broadcastResume(catalog, name, "layer_conflict");
+ return badRequest(lce.applyStatus(), catalog, name, lce.getMessage());
} catch (final Throwable t) {
+ // Some exceptions wrap LayerConflictException as cause; unwrap so the operator
+ // still sees the structured 400 response. We probe to a small depth to avoid
+ // recursing through pathological cause chains.
+ final LayerConflictException wrapped = findLayerConflict(t);
+ if (wrapped != null) {
+ log.warn("runtime-rule STRUCTURAL apply rejected for {}/{} — layer conflict "
+ + "(wrapped): {}", catalog, name, wrapped.getMessage());
+ dslManager.getSuspendCoord().localResume(catalog, name);
+ broadcastResume(catalog, name, "layer_conflict");
+ return badRequest(wrapped.applyStatus(), catalog, name, wrapped.getMessage());
+ }
log.error("runtime-rule STRUCTURAL apply threw for {}/{}", catalog, name, t);
dslManager.getSuspendCoord().localResume(catalog, name);
// Peers went SUSPENDED on our earlier broadcast; let them know the apply
@@ -1741,6 +1789,24 @@ public class RuntimeRuleService {
jsonBody(applyStatus, catalog, name, message));
}
+ /**
+ * Walk up to 8 levels of cause chain looking for a {@link LayerConflictException}. The
+ * DSL engines wrap exceptions in {@code EngineCompileException} / {@code ApplyException},
+ * which in turn may be wrapped by infrastructure exceptions; surfacing the conflict's
+ * structured status / message regardless of wrapping depth keeps the operator-facing
+ * envelope consistent with the design intent that conflicts ARE bad-request shaped.
+ */
+ private static LayerConflictException findLayerConflict(final Throwable top) {
+ Throwable cursor = top;
+ for (int i = 0; cursor != null && i < 8; i++) {
+ if (cursor instanceof LayerConflictException) {
+ return (LayerConflictException) cursor;
+ }
+ cursor = cursor.getCause();
+ }
+ return null;
+ }
+
private static HttpResponse serverError(final String applyStatus, final String catalog,
final String name, final String message) {
return HttpResponse.of(HttpStatus.INTERNAL_SERVER_ERROR, MediaType.JSON_UTF_8,
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/state/EngineApplied.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/state/EngineApplied.java
index c2fe3f49fa..8e9c269380 100644
--- a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/state/EngineApplied.java
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/state/EngineApplied.java
@@ -20,6 +20,7 @@ package org.apache.skywalking.oap.server.receiver.runtimerule.state;
import java.util.Set;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.AppliedClaims;
/**
* Engine-opaque per-rule applied artefact slot on {@link AppliedRuleScript}. Every engine's
@@ -93,4 +94,20 @@ public interface EngineApplied {
* @return immutable, possibly empty set of metric names
*/
Set<String> alarmResetTargets();
+
+ /**
+ * Snapshot of the {@link AppliedClaims} produced by this apply's interaction with the
+ * runtime-layer registry, or {@code null} when the apply declared no
+ * {@code layerDefinitions:} (the common case for bundled / non-layer rules). The
+ * structural commit coordinator calls
+ * {@link org.apache.skywalking.oap.server.receiver.runtimerule.layer.RuntimeLayerRegistry#rollback}
+ * with this token on a persist / commit-tail failure that lands outside the applier's
+ * own try/catch.
+ *
+ * <p>Default returns {@code null} so existing engines (and any future engines that
+ * do not own layer state) need no implementation change.
+ */
+ default AppliedClaims appliedLayerClaims() {
+ return null;
+ }
}
diff --git a/oap-server/server-admin/runtime-rule/src/test/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/MalFileApplierTest.java b/oap-server/server-admin/runtime-rule/src/test/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/MalFileApplierTest.java
index 726b1751ae..dc4222b491 100644
--- a/oap-server/server-admin/runtime-rule/src/test/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/MalFileApplierTest.java
+++ b/oap-server/server-admin/runtime-rule/src/test/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/MalFileApplierTest.java
@@ -22,8 +22,11 @@ import java.util.Collections;
import java.util.HashSet;
import java.util.Set;
import javassist.ClassPool;
+import org.apache.skywalking.oap.server.core.analysis.Layer;
import org.apache.skywalking.oap.server.core.analysis.meter.MeterSystem;
import org.apache.skywalking.oap.server.core.storage.model.StorageManipulationOpt;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.LayerConflictException;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.RuntimeLayerRegistry;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.mockito.Mockito;
@@ -207,6 +210,84 @@ class MalFileApplierTest {
assertEquals(setOf("meter_x_one"), applied.getRegisteredMetricNames());
}
+ @Test
+ void applyWithLayerDefinitionsRegistersLayerAndCarriesAppliedClaims() throws Exception {
+ // Happy path: a runtime MAL rule that declares a new layer in layerDefinitions
+ // with a properly pinned ordinal in the runtime tier (>=100_000) must:
+ // (a) register the layer through the dynamic channel (Layer.isDynamic = true);
+ // (b) carry the AppliedClaims token on the returned Applied artifact so the
+ // structural commit coordinator can roll it back on persist failure;
+ // (c) populate the layer-registry refcount with this rule as the claimant.
+ // Uses an isolated RuntimeLayerRegistry to avoid touching JVM-wide state.
+ // `normal:` is intentionally omitted to exercise the LayerDefinition default
+ // (true) — same operator ergonomics the bundled yaml + layer-extensions tier
+ // already enjoy.
+ final RuntimeLayerRegistry isolatedRegistry = new RuntimeLayerRegistry();
+ final MalFileApplier layerApplier = new MalFileApplier(meterSystem, isolatedRegistry);
+ final String yaml =
+ "metricPrefix: meter_hl\n"
+ + "layerDefinitions:\n"
+ + " - name: MAL_APPLIER_HAPPY_LAYER\n"
+ + " ordinal: 100600\n"
+ + "expSuffix: service(['host'], Layer.OS_LINUX)\n"
+ + "metricsRules:\n"
+ + " - name: ok\n"
+ + " exp: m.sum(['host'])\n";
+ final MalFileApplier.Applied applied =
+ layerApplier.apply(yaml, "otel-rules/happy_layer_mal", "hashHL");
+ try {
+ // (a) layer registered through the dynamic channel with the default normal
+ assertEquals(100_600, Layer.nameOf("MAL_APPLIER_HAPPY_LAYER").value());
+ assertTrue(Layer.nameOf("MAL_APPLIER_HAPPY_LAYER").isNormal(),
+ "omitted normal: defaults to true, same as the other tiers");
+ assertTrue(Layer.isDynamic("MAL_APPLIER_HAPPY_LAYER"));
+ // (b) AppliedClaims carried for orchestrator rollback
+ assertNotNull(applied.appliedLayerClaims());
+ assertTrue(applied.appliedLayerClaims().getCurrentLayerNames()
+ .contains("MAL_APPLIER_HAPPY_LAYER"));
+ assertTrue(applied.appliedLayerClaims().getNewlyRegistered()
+ .contains("MAL_APPLIER_HAPPY_LAYER"));
+ // (c) refcount tracks this rule as the claimant
+ assertTrue(isolatedRegistry.snapshot().get("MAL_APPLIER_HAPPY_LAYER")
+ .contains("otel-rules/happy_layer_mal"));
+ } finally {
+ // Clean up so JVM-wide Layer state stays sane for sibling tests.
+ isolatedRegistry.removeRule("otel-rules/happy_layer_mal");
+ }
+ }
+
+ @Test
+ void applyWithLayerOrdinalBelowFloorThrowsLayerConflict() {
+ // Sad path mirror of the happy-path test: ordinal=99_999 (the last ordinal of
+ // the boot-time external tier — one slot below the 100_000 runtime floor) must
+ // be rejected with layer_ordinal_out_of_range. The boundary value is the most
+ // realistic operator mistake: the operator remembers "external layers go in
+ // 10_000-99_999" and reuses that convention for a runtime rule. The applier
+ // wraps the LayerConflictException through; the REST handler maps it to HTTP
+ // 400 with the structured envelope. This proves the validation gate fires
+ // BEFORE compile so no MeterSystem.create runs for a rule we're going to
+ // reject.
+ final RuntimeLayerRegistry isolatedRegistry = new RuntimeLayerRegistry();
+ final MalFileApplier layerApplier = new MalFileApplier(meterSystem, isolatedRegistry);
+ final String yaml =
+ "metricPrefix: meter_bad_floor\n"
+ + "layerDefinitions:\n"
+ + " - name: MAL_APPLIER_BAD_FLOOR\n"
+ + " ordinal: 99999\n"
+ + "expSuffix: service(['host'], Layer.OS_LINUX)\n"
+ + "metricsRules:\n"
+ + " - name: ok\n"
+ + " exp: m.sum(['host'])\n";
+ final LayerConflictException ex = assertThrows(
+ LayerConflictException.class,
+ () -> layerApplier.apply(yaml, "otel-rules/bad_floor", "h"));
+ assertEquals(LayerConflictException.Status.LAYER_ORDINAL_OUT_OF_RANGE, ex.getStatus());
+ // MeterSystem.create was never invoked because validation fired first.
+ verify(meterSystem, Mockito.never())
+ .create(anyString(), anyString(), any(), any(ClassPool.class), any(ClassLoader.class),
+ any(StorageManipulationOpt.class));
+ }
+
@Test
void legacyTwoArgApplyUsesEmptyHash() throws Exception {
// Back-compat overload — loader identity still constructible, just with a less
diff --git a/oap-server/server-admin/runtime-rule/src/test/java/org/apache/skywalking/oap/server/receiver/runtimerule/extension/DbOverrideRuntimeRuleResolverTest.java b/oap-server/server-admin/runtime-rule/src/test/java/org/apache/skywalking/oap/server/receiver/runtimerule/extension/DbOverrideRuntimeRuleResolverTest.java
new file mode 100644
index 0000000000..59a07958ad
--- /dev/null
+++ b/oap-server/server-admin/runtime-rule/src/test/java/org/apache/skywalking/oap/server/receiver/runtimerule/extension/DbOverrideRuntimeRuleResolverTest.java
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.receiver.runtimerule.extension;
+
+import java.lang.reflect.Method;
+import java.nio.charset.StandardCharsets;
+import org.apache.skywalking.oap.server.core.rule.ext.StaticRuleRegistry;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Covers the layer-override-forbidden guard {@link DbOverrideRuntimeRuleResolver}
+ * applies to runtime overrides of bundled MAL/LAL rules. The guard exists because
+ * bundled and runtime are separate ownership channels for layers; the runtime path
+ * must not overwrite the bundled channel.
+ */
+class DbOverrideRuntimeRuleResolverTest {
+
+ @BeforeEach
+ @AfterEach
+ void clearRegistry() throws Exception {
+ // StaticRuleRegistry is process-wide; isolate from sibling tests.
+ final Method m = StaticRuleRegistry.class.getDeclaredMethod("clear");
+ m.setAccessible(true);
+ m.invoke(StaticRuleRegistry.active());
+ }
+
+ @Test
+ void overrideWithoutLayerDefinitionsIsAllowed() {
+ StaticRuleRegistry.active().record(
+ "otel-rules", "no_layers_bundled",
+ "metricPrefix: foo\nmetricsRules:\n - name: x\n exp: a.sum(['s'])\n"
+ .getBytes(StandardCharsets.UTF_8));
+ final byte[] runtimeBody =
+ "metricPrefix: foo\nmetricsRules:\n - name: x\n exp: a.sum(['s'])\n"
+ .getBytes(StandardCharsets.UTF_8);
+ assertFalse(DbOverrideRuntimeRuleResolver.isLayerOverrideOnBundled(
+ "otel-rules", "no_layers_bundled", runtimeBody));
+ }
+
+ @Test
+ void overrideWithLayerDefinitionsOnDiskTwinIsForbidden() {
+ // Disk twin exists.
+ StaticRuleRegistry.active().record(
+ "otel-rules", "with_layers_bundled",
+ "metricPrefix: foo\nmetricsRules:\n - name: x\n exp: a.sum(['s'])\n"
+ .getBytes(StandardCharsets.UTF_8));
+ // Runtime override adds layerDefinitions — the case the resolver must drop.
+ final String override =
+ "metricPrefix: foo\n"
+ + "layerDefinitions:\n"
+ + " - name: TEST_RT_OVERRIDE_LAYER\n"
+ + " ordinal: 100700\n"
+ + "metricsRules:\n - name: x\n exp: a.sum(['s'])\n";
+ assertTrue(DbOverrideRuntimeRuleResolver.isLayerOverrideOnBundled(
+ "otel-rules", "with_layers_bundled",
+ override.getBytes(StandardCharsets.UTF_8)));
+ }
+
+ @Test
+ void pureRuntimeRuleWithLayerDefinitionsIsNotForbidden() {
+ // No disk twin recorded — pure runtime rule. layerDefinitions are allowed and
+ // go through the dynamic channel via RuleSync.runOnce. The guard returns false.
+ final String pureRuntime =
+ "metricPrefix: foo\n"
+ + "layerDefinitions:\n"
+ + " - name: TEST_PURE_RUNTIME_LAYER\n"
+ + " ordinal: 100701\n"
+ + "metricsRules:\n - name: x\n exp: a.sum(['s'])\n";
+ assertFalse(DbOverrideRuntimeRuleResolver.isLayerOverrideOnBundled(
+ "otel-rules", "no_disk_twin_pure_runtime",
+ pureRuntime.getBytes(StandardCharsets.UTF_8)));
+ }
+
+ @Test
+ void unknownCatalogReturnsFalse() {
+ // Catalog the guard doesn't know how to parse — returns false (apply will fail
+ // its own validation later if the content is malformed).
+ assertFalse(DbOverrideRuntimeRuleResolver.isLayerOverrideOnBundled(
+ "unknown-catalog", "x",
+ "anything".getBytes(StandardCharsets.UTF_8)));
+ }
+
+ @Test
+ void malformedYamlReturnsFalse() {
+ StaticRuleRegistry.active().record(
+ "otel-rules", "malformed_check",
+ "metricPrefix: foo\n".getBytes(StandardCharsets.UTF_8));
+ // Garbage bytes: the guard returns false rather than throwing. The apply path
+ // surfaces the YAML error with a clearer message; the override-forbidden guard
+ // is not the right place to flag parse failures.
+ assertFalse(DbOverrideRuntimeRuleResolver.isLayerOverrideOnBundled(
+ "otel-rules", "malformed_check",
+ "this: is: not: valid: yaml: at: all".getBytes(StandardCharsets.UTF_8)));
+ }
+}
diff --git a/oap-server/server-admin/runtime-rule/src/test/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/RuntimeLayerRegistryTest.java b/oap-server/server-admin/runtime-rule/src/test/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/RuntimeLayerRegistryTest.java
new file mode 100644
index 0000000000..2b595145a9
--- /dev/null
+++ b/oap-server/server-admin/runtime-rule/src/test/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/RuntimeLayerRegistryTest.java
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.receiver.runtimerule.layer;
+
+import java.util.Arrays;
+import java.util.Collections;
+import org.apache.skywalking.oap.server.core.analysis.Layer;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Exercises {@link RuntimeLayerRegistry}'s refcount semantics: net-new register/unregister
+ * on first/last claim, idempotent re-claim by the same rule, multi-rule sharing, and the
+ * apply/rollback round-trip used by the structural commit coordinator.
+ *
+ * <p>Uses ordinals in {@code [200_000, 200_999]} so collisions with sibling tests and any
+ * boot-time registrations cannot occur. Cleanup in {@link #cleanup} unregisters every
+ * dynamic layer it touched so JVM-shared state stays clean for other test classes.
+ */
+class RuntimeLayerRegistryTest {
+
+ private static final String RULE_A = "otel-rules/test-a";
+ private static final String RULE_B = "otel-rules/test-b";
+
+ private final RuntimeLayerRegistry registry = new RuntimeLayerRegistry();
+
+ @AfterEach
+ void cleanup() {
+ // Drop every claim, which cascades into Layer.unregisterDynamic for any net-zero
+ // layer. Defensive: walk a snapshot so we don't mutate the map mid-iteration.
+ for (final String rule : Arrays.asList(RULE_A, RULE_B, "otel-rules/test-c")) {
+ registry.removeRule(rule);
+ }
+ }
+
+ @Test
+ void applyRegistersNewLayerAndCountsClaim() {
+ final LayerClaim claim = new LayerClaim("DYN_REGISTRY_A", 200_000, true);
+ final AppliedClaims applied = registry.apply(RULE_A, Collections.singletonList(claim));
+
+ assertTrue(applied.getNewlyRegistered().contains("DYN_REGISTRY_A"));
+ assertEquals(Collections.singleton("DYN_REGISTRY_A"), registry.claimsOf(RULE_A));
+ assertEquals(200_000, Layer.nameOf("DYN_REGISTRY_A").value());
+ }
+
+ @Test
+ void secondClaimOnSameLayerIsIdempotent() {
+ final LayerClaim claim = new LayerClaim("DYN_REGISTRY_SHARED", 200_001, true);
+ final AppliedClaims firstApplied = registry.apply(RULE_A, Collections.singletonList(claim));
+ final AppliedClaims secondApplied = registry.apply(RULE_B, Collections.singletonList(claim));
+
+ // First apply registers the layer; second only adds B as a claimant.
+ assertTrue(firstApplied.getNewlyRegistered().contains("DYN_REGISTRY_SHARED"));
+ assertTrue(secondApplied.getNewlyRegistered().isEmpty());
+
+ // Both A and B are claimants.
+ assertEquals(2, registry.snapshot().get("DYN_REGISTRY_SHARED").size());
+ }
+
+ @Test
+ void unregisterDropsLayerWhenLastClaimGone() {
+ final LayerClaim claim = new LayerClaim("DYN_REGISTRY_DROP", 200_002, true);
+ registry.apply(RULE_A, Collections.singletonList(claim));
+ registry.apply(RULE_B, Collections.singletonList(claim));
+
+ // Removing A keeps the layer alive (B still claims it).
+ registry.removeRule(RULE_A);
+ assertEquals(200_002, Layer.nameOf("DYN_REGISTRY_DROP").value());
+
+ // Removing B drops the last claim — layer goes away.
+ registry.removeRule(RULE_B);
+ assertSame(Layer.UNDEFINED, Layer.nameOf("DYN_REGISTRY_DROP"));
+ }
+
+ @Test
+ void updateReplacesPriorClaimsAndUnregistersOrphans() {
+ final LayerClaim original = new LayerClaim("DYN_REGISTRY_ORPHAN", 200_003, true);
+ final LayerClaim replacement = new LayerClaim("DYN_REGISTRY_KEEP", 200_004, true);
+
+ registry.apply(RULE_A, Collections.singletonList(original));
+ assertEquals(200_003, Layer.nameOf("DYN_REGISTRY_ORPHAN").value());
+
+ registry.apply(RULE_A, Collections.singletonList(replacement));
+
+ // The orphaned layer is gone (no other rule claims it).
+ assertSame(Layer.UNDEFINED, Layer.nameOf("DYN_REGISTRY_ORPHAN"));
+ // The replacement is now registered and claimed by RULE_A.
+ assertEquals(200_004, Layer.nameOf("DYN_REGISTRY_KEEP").value());
+ assertEquals(Collections.singleton("DYN_REGISTRY_KEEP"), registry.claimsOf(RULE_A));
+ }
+
+ @Test
+ void rollbackRestoresPriorStateAfterAppliedUpdate() {
+ final LayerClaim original = new LayerClaim("DYN_REGISTRY_ROLLBACK_PRIOR", 200_005, true);
+ final LayerClaim replacement = new LayerClaim("DYN_REGISTRY_ROLLBACK_NEW", 200_006, true);
+
+ registry.apply(RULE_A, Collections.singletonList(original));
+ final AppliedClaims applied = registry.apply(RULE_A, Collections.singletonList(replacement));
+
+ // Confirm intermediate state.
+ assertSame(Layer.UNDEFINED, Layer.nameOf("DYN_REGISTRY_ROLLBACK_PRIOR"));
+ assertEquals(200_006, Layer.nameOf("DYN_REGISTRY_ROLLBACK_NEW").value());
+
+ registry.rollback(applied);
+
+ // Prior layer restored; replacement unregistered.
+ assertEquals(200_005, Layer.nameOf("DYN_REGISTRY_ROLLBACK_PRIOR").value());
+ assertSame(Layer.UNDEFINED, Layer.nameOf("DYN_REGISTRY_ROLLBACK_NEW"));
+ assertEquals(Collections.singleton("DYN_REGISTRY_ROLLBACK_PRIOR"),
+ registry.claimsOf(RULE_A));
+ }
+
+ @Test
+ void selfUpdateWithDifferentOrdinalIsPermitted() {
+ // Rule edits its OWN previously-declared layer — conflict checker treats the
+ // prior claim as already-removed, so a triple change is allowed.
+ final LayerClaim before = new LayerClaim("DYN_REGISTRY_SELF_EDIT", 200_007, true);
+ final LayerClaim after = new LayerClaim("DYN_REGISTRY_SELF_EDIT", 200_008, false);
+
+ registry.apply(RULE_A, Collections.singletonList(before));
+ registry.apply(RULE_A, Collections.singletonList(after));
+
+ final Layer current = Layer.nameOf("DYN_REGISTRY_SELF_EDIT");
+ assertEquals(200_008, current.value());
+ assertEquals(false, current.isNormal());
+ }
+
+ @Test
+ void conflictAcrossRulesRaisesLayerNameConflict() {
+ registry.apply(RULE_A, Collections.singletonList(
+ new LayerClaim("DYN_REGISTRY_CROSS_RULE", 200_009, true)));
+
+ // Rule B declares same name but different ordinal — must be rejected.
+ final LayerConflictException ex = assertThrows(
+ LayerConflictException.class,
+ () -> registry.apply(RULE_B, Collections.singletonList(
+ new LayerClaim("DYN_REGISTRY_CROSS_RULE", 200_010, true))));
+
+ assertEquals(LayerConflictException.Status.LAYER_NAME_CONFLICT, ex.getStatus());
+ assertEquals("layer_name_conflict", ex.applyStatus());
+ assertTrue(ex.getMessage().contains("DYN_REGISTRY_CROSS_RULE"));
+ assertTrue(ex.getMessage().contains(RULE_A), "message must name the source rule");
+ assertTrue(ex.getMessage().contains(RULE_B), "message must name the offender");
+ }
+
+ @Test
+ void conflictRaisesOrdinalCollisionForDifferentNameSameOrdinal() {
+ registry.apply(RULE_A, Collections.singletonList(
+ new LayerClaim("DYN_REGISTRY_ORD_A", 200_011, true)));
+
+ final LayerConflictException ex = assertThrows(
+ LayerConflictException.class,
+ () -> registry.apply(RULE_B, Collections.singletonList(
+ new LayerClaim("DYN_REGISTRY_ORD_B", 200_011, true))));
+
+ assertEquals(LayerConflictException.Status.LAYER_ORDINAL_COLLISION, ex.getStatus());
+ assertEquals("layer_ordinal_collision", ex.applyStatus());
+ assertTrue(ex.getMessage().contains("200011") || ex.getMessage().contains("200_011"));
+ }
+
+ @Test
+ void conflictRaisesOrdinalOutOfRangeBelowFloor() {
+ final LayerConflictException ex = assertThrows(
+ LayerConflictException.class,
+ () -> registry.apply(RULE_A, Collections.singletonList(
+ new LayerClaim("DYN_REGISTRY_LOW", 99_999, true))));
+
+ assertEquals(LayerConflictException.Status.LAYER_ORDINAL_OUT_OF_RANGE, ex.getStatus());
+ assertTrue(ex.getMessage().contains("99999"));
+ assertTrue(ex.getMessage().contains("100000"));
+ }
+
+ @Test
+ void conflictRaisesNameInvalidForBadShape() {
+ final LayerConflictException ex = assertThrows(
+ LayerConflictException.class,
+ () -> registry.apply(RULE_A, Collections.singletonList(
+ new LayerClaim("bad-name", 200_012, true))));
+
+ assertEquals(LayerConflictException.Status.LAYER_NAME_INVALID, ex.getStatus());
+ assertTrue(ex.getMessage().contains("bad-name"));
+ }
+
+ @Test
+ void conflictAgainstBundledLayerLabelsSourceClearly() {
+ // MESH is the built-in layer with ordinal 1 — declaring it with a runtime
+ // ordinal yields a name conflict; the source must be labelled "built-in or
+ // boot-time extension" so the operator knows it can't be aligned at all.
+ final LayerConflictException ex = assertThrows(
+ LayerConflictException.class,
+ () -> registry.apply(RULE_A, Collections.singletonList(
+ new LayerClaim("MESH", 200_013, true))));
+
+ assertEquals(LayerConflictException.Status.LAYER_NAME_CONFLICT, ex.getStatus());
+ assertTrue(ex.getMessage().contains("MESH"));
+ assertTrue(ex.getMessage().contains("built-in"),
+ "message must label the source as built-in: " + ex.getMessage());
+ }
+
+ @Test
+ void inBatchDuplicateNameWithDifferentTriplesIsRejected() {
+ // A single batch declaring the same name twice with different triples can never
+ // be satisfied — must be caught BEFORE any mutation lands so apply stays atomic.
+ final LayerConflictException ex = assertThrows(
+ LayerConflictException.class,
+ () -> registry.apply(RULE_A, Arrays.asList(
+ new LayerClaim("DYN_BATCH_DUP_NAME", 200_020, true),
+ new LayerClaim("DYN_BATCH_DUP_NAME", 200_021, true))));
+ assertEquals(LayerConflictException.Status.LAYER_NAME_CONFLICT, ex.getStatus());
+ // Neither variant should have been registered — the registry must look unchanged.
+ assertSame(Layer.UNDEFINED,
+ Layer.nameOf("DYN_BATCH_DUP_NAME"));
+ assertTrue(registry.claimsOf(RULE_A).isEmpty());
+ }
+
+ @Test
+ void inBatchDuplicateOrdinalUnderDifferentNamesIsRejected() {
+ final LayerConflictException ex = assertThrows(
+ LayerConflictException.class,
+ () -> registry.apply(RULE_A, Arrays.asList(
+ new LayerClaim("DYN_BATCH_ORD_X", 200_022, true),
+ new LayerClaim("DYN_BATCH_ORD_Y", 200_022, true))));
+ assertEquals(LayerConflictException.Status.LAYER_ORDINAL_COLLISION, ex.getStatus());
+ // Neither name registered.
+ assertSame(Layer.UNDEFINED,
+ Layer.nameOf("DYN_BATCH_ORD_X"));
+ assertSame(Layer.UNDEFINED,
+ Layer.nameOf("DYN_BATCH_ORD_Y"));
+ }
+
+ @Test
+ void inBatchSameTripleRepeatedIsAccepted() {
+ // Idempotent duplicate within a batch (operator copy-paste, or a generator that
+ // emits the same entry twice) is harmless and must not be rejected.
+ registry.apply(RULE_A, Arrays.asList(
+ new LayerClaim("DYN_BATCH_IDEMPOTENT", 200_023, true),
+ new LayerClaim("DYN_BATCH_IDEMPOTENT", 200_023, true)));
+ assertEquals(200_023,
+ Layer
+ .nameOf("DYN_BATCH_IDEMPOTENT").value());
+ }
+
+ @Test
+ void selfUpdateSwappingOrdinalsAcrossTwoLayersIsAtomic() {
+ // Rule R declares LAYER_X@200_030 and LAYER_Y@200_031. Operator pushes a swap:
+ // LAYER_X@200_031, LAYER_Y@200_030. validate must permit it (each is a self-claim
+ // freeing the other's ordinal), apply must execute the swap without intermediate
+ // ordinal collision throwing — that's the bug finding 2 surfaced for the per-claim
+ // loop. The two-pass apply unregisters all prior claims first, then registers all
+ // new triples, so the freed slots are available for the swap.
+ registry.apply(RULE_A, Arrays.asList(
+ new LayerClaim("DYN_SWAP_X", 200_030, true),
+ new LayerClaim("DYN_SWAP_Y", 200_031, true)));
+ registry.apply(RULE_A, Arrays.asList(
+ new LayerClaim("DYN_SWAP_X", 200_031, true),
+ new LayerClaim("DYN_SWAP_Y", 200_030, true)));
+
+ assertEquals(200_031,
+ Layer
+ .nameOf("DYN_SWAP_X").value());
+ assertEquals(200_030,
+ Layer
+ .nameOf("DYN_SWAP_Y").value());
+ }
+
+ @Test
+ void softClaimAgainstBundledLayerSucceedsButCannotRemove() {
+ // Pre-register a layer through the bundled channel (Layer.register) at an
+ // ordinal in the runtime range. This simulates a boot-time external entry
+ // (layer-extensions.yml / bundled MAL/LAL) that — against the recommended
+ // tier convention — happens to live at >=100_000. A runtime rule claiming the
+ // SAME triple gets a soft claim: refcount tracks the rule, Layer stays bundled,
+ // unregisterDynamic refuses on rule delete.
+ Layer.register("DYN_REG_SOFT_CLAIM_TARGET", 250_010, true);
+ try {
+ registry.apply(RULE_A, Collections.singletonList(
+ new LayerClaim("DYN_REG_SOFT_CLAIM_TARGET", 250_010, true)));
+
+ assertTrue(registry.snapshot().get("DYN_REG_SOFT_CLAIM_TARGET").contains(RULE_A));
+ assertFalse(Layer.isDynamic("DYN_REG_SOFT_CLAIM_TARGET"),
+ "soft claim does not flip the channel from bundled to dynamic");
+ assertEquals(250_010, Layer.nameOf("DYN_REG_SOFT_CLAIM_TARGET").value());
+
+ registry.removeRule(RULE_A);
+
+ assertEquals(250_010, Layer.nameOf("DYN_REG_SOFT_CLAIM_TARGET").value(),
+ "removing the soft-claimant rule must NOT unregister the bundled layer");
+ assertFalse(registry.snapshot().containsKey("DYN_REG_SOFT_CLAIM_TARGET"));
+ } finally {
+ // Cannot unregister via the dynamic path (it's bundled). Layer stays in the
+ // JVM-wide registry but the unique name prevents sibling-test interference.
+ }
+ }
+
+ @Test
+ void selfUpdateReusingMultiClaimantOrdinalIsRejectedStructured() {
+ // Rules A + B both claim OLD@200_090. A then tries to update to NEW@200_090 —
+ // reusing the ordinal of its own prior layer under a new name. Without the
+ // multi-claimant check, validate would pass (A held OLD priorly), apply's Pass-1
+ // would only drop A's claim from OLD (B still holds it), and registerDynamic
+ // would throw an unstructured ordinal collision deep in apply. The structured
+ // layer_ordinal_collision must surface from validate before any mutation lands.
+ final LayerClaim shared = new LayerClaim("DYN_MULTI_CLAIM_OLD", 200_090, true);
+ registry.apply(RULE_A, Collections.singletonList(shared));
+ registry.apply(RULE_B, Collections.singletonList(shared));
+
+ final LayerConflictException ex = assertThrows(
+ LayerConflictException.class,
+ () -> registry.apply(RULE_A, Collections.singletonList(
+ new LayerClaim("DYN_MULTI_CLAIM_NEW", 200_090, true))));
+ assertEquals(LayerConflictException.Status.LAYER_ORDINAL_COLLISION, ex.getStatus());
+ assertTrue(ex.getMessage().contains(RULE_B),
+ "message must name the rule still holding the prior layer");
+ // State unchanged: OLD still owned by A + B, no NEW.
+ assertSame(Layer.UNDEFINED, Layer.nameOf("DYN_MULTI_CLAIM_NEW"));
+ assertEquals(200_090, Layer.nameOf("DYN_MULTI_CLAIM_OLD").value());
+ }
+
+ @Test
+ void reactivationAfterInactiveSurfacesConflictAgainstNewClaimant() {
+ // Rule A claims a dynamic layer. A is then "inactivated" (its claims removed
+ // from the registry). While A is inactive, rule B starts claiming the same
+ // name at a different ordinal. When the operator reactivates A, the apply
+ // must surface the conflict so the operator can edit or drop A.
+ registry.apply(RULE_A, Collections.singletonList(
+ new LayerClaim("DYN_REACTIVATE_PROBE", 200_080, true)));
+ // Inactivate A.
+ registry.removeRule(RULE_A);
+ assertSame(Layer.UNDEFINED, Layer.nameOf("DYN_REACTIVATE_PROBE"));
+
+ // Meanwhile, B claims the same name at a different ordinal.
+ registry.apply(RULE_B, Collections.singletonList(
+ new LayerClaim("DYN_REACTIVATE_PROBE", 200_081, true)));
+
+ // Operator pushes A again (reactivation). The original triple no longer fits.
+ final LayerConflictException ex = assertThrows(
+ LayerConflictException.class,
+ () -> registry.apply(RULE_A, Collections.singletonList(
+ new LayerClaim("DYN_REACTIVATE_PROBE", 200_080, true))));
+ assertEquals(LayerConflictException.Status.LAYER_NAME_CONFLICT, ex.getStatus());
+ assertTrue(ex.getMessage().contains(RULE_B),
+ "message must name the currently-active claimant so operator knows what to align with");
+ }
+
+ @Test
+ void noOpRedeclareDoesNotPerturbLiveLayer() {
+ // Re-applying the SAME triple by the SAME rule should be a no-op at the Layer
+ // level (no unregister + re-register churn that briefly leaves the layer
+ // unresolvable). Confirmed by re-checking the identity of the Layer instance
+ // before and after the second apply.
+ registry.apply(RULE_A, Collections.singletonList(
+ new LayerClaim("DYN_STABLE_REAPPLY", 200_040, true)));
+ final Layer before =
+ Layer.nameOf("DYN_STABLE_REAPPLY");
+ registry.apply(RULE_A, Collections.singletonList(
+ new LayerClaim("DYN_STABLE_REAPPLY", 200_040, true)));
+ final Layer after =
+ Layer.nameOf("DYN_STABLE_REAPPLY");
+ assertSame(before, after, "same-triple re-apply must not churn the Layer instance");
+ }
+}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java
index 0d7e1a8354..bb702d5c1a 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java
@@ -20,6 +20,7 @@ package org.apache.skywalking.oap.server.core.analysis;
import java.util.Comparator;
import java.util.Map;
+import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
@@ -35,35 +36,69 @@ import org.apache.skywalking.oap.server.core.UnexpectedException;
* {@link #valueOf(String)}) or by ordinal ({@link #valueOf(int)}); {@link #values()} returns
* every registered layer sorted by ordinal.
*
- * <p>Layers are persisted by ordinal (int). Built-in ordinals 0–49 are frozen forever and
- * must never be reused. Ordinals 50–999 are reserved by convention for future built-ins;
- * external extensions are <strong>recommended</strong> to start at {@code 1000} to avoid
- * colliding with future built-ins on OAP upgrade. The recommendation is informational and
- * not enforced — any non-colliding ordinal is accepted — but a future built-in landing on
- * the same ordinal as an extension layer will cause OAP boot to fail loudly via the
- * ordinal uniqueness check, which is the upgrade-time detection mechanism.
+ * <p>Layers are persisted by ordinal (int). The ordinal space is partitioned by tier so
+ * each registration channel has a non-overlapping range:
+ * <ul>
+ * <li>{@code 0 – 9_999} — built-in {@code Layer.*} constants. Currently {@code 0..49} in
+ * use; the rest of the range is reserved for future built-ins. Ordinals already
+ * persisted in storage are frozen forever and must never be reused.</li>
+ * <li>{@code 10_000 – 99_999} — boot-time external layers: {@code layer-extensions.yml},
+ * {@code LayerExtension} SPI, and bundled MAL/LAL {@code layerDefinitions:} blocks.
+ * Registered through {@link #register} before {@link #seal()}.</li>
+ * <li>{@code 100_000 – Integer.MAX_VALUE} — runtime DSL dynamic layers introduced by
+ * MAL/LAL hot-update {@code layerDefinitions:} blocks. Registered through
+ * {@link #registerDynamic} after {@link #seal()} and removable through
+ * {@link #unregisterDynamic} when the last declaring rule is removed.</li>
+ * </ul>
+ * The tier boundaries are enforced for {@link #registerDynamic} / {@link #unregisterDynamic}
+ * (ordinal must be {@code >= 100_000}). {@link #register} accepts any non-colliding ordinal
+ * for backward compatibility, but operator-managed extensions <strong>should</strong> use
+ * the {@code 10_000+} tier to leave the {@code 100_000+} space for runtime DSL.
*
- * <p>Registration paths (all funnel through {@link #register}):
+ * <p>Registration paths (all funnel through {@link #register} or {@link #registerDynamic}):
* <ol>
* <li>Operator yaml — {@code layer-extensions.yml} on the OAP classpath/config dir.</li>
* <li>Java SPI — {@code LayerExtension} discovered via {@code ServiceLoader}.</li>
- * <li>DSL inline — MAL/LAL rule files declaring a top-level {@code layerDefinitions:}
- * block; the DSL loader funnels each entry through this method before compiling.</li>
+ * <li>DSL inline (boot) — MAL/LAL rule files declaring a top-level {@code layerDefinitions:}
+ * block; the DSL loader funnels each entry through {@link #register} before compiling.</li>
+ * <li>DSL inline (runtime) — runtime-rule MAL/LAL hot-update files declaring
+ * {@code layerDefinitions:}; funneled through {@link #registerDynamic}. The runtime-rule
+ * module's {@code RuntimeLayerRegistry} refcounts these per declaring rule file and
+ * invokes {@link #unregisterDynamic} when the last claim drops.</li>
* </ol>
* The registry is sealed at the start of {@code CoreModuleProvider.notifyAfterCompleted()};
- * any registration attempt after that throws.
+ * after seal, only {@link #registerDynamic} and {@link #unregisterDynamic} can mutate the
+ * registry.
*/
public final class Layer {
private static final Pattern NAME_PATTERN = Pattern.compile("[A-Z][A-Z0-9_]*");
+ /**
+ * Lowest ordinal allowed for runtime DSL dynamic layers. Ordinals below this value are
+ * reserved for built-in {@code Layer.*} constants ({@code 0–9_999}) and boot-time
+ * external extensions ({@code 10_000–99_999}). See class javadoc for the full tier
+ * convention.
+ */
+ public static final int RUNTIME_DYNAMIC_MIN_ORDINAL = 100_000;
+
private static final Map<Integer, Layer> BY_VALUE = new ConcurrentHashMap<>();
private static final Map<String, Layer> BY_NAME = new ConcurrentHashMap<>();
+ /**
+ * Explicit ownership marker: every name added through {@link #registerDynamic} lands
+ * here, and {@link #unregisterDynamic} consults this set to refuse removing layers
+ * that came through the boot-time {@link #register} channel. Distinguishing by ordinal
+ * range alone (the original heuristic) was unsafe — operator yaml or bundled MAL/LAL
+ * can land in the {@code >=100_000} range and would have been wrongly considered
+ * removable. Adding here is idempotent on re-registration.
+ */
+ private static final Set<String> DYNAMIC_NAMES = ConcurrentHashMap.newKeySet();
private static volatile boolean SEALED = false;
/**
- * Snapshot of all registered layers sorted by ordinal, frozen at {@link #seal()}.
- * Before seal, {@link #values()} computes on the fly so newly registered layers are
- * visible; after seal, {@link #values()} returns a clone of this cached array.
+ * Snapshot of all registered layers sorted by ordinal. Populated once at {@link #seal()};
+ * after seal, {@link #registerDynamic} / {@link #unregisterDynamic} rebuild this array
+ * so {@link #values()} reflects post-seal mutations. Before seal, {@link #values()}
+ * throws because the registry may still grow via boot-time {@link #register} calls.
*/
private static volatile Layer[] CACHED_VALUES = null;
@@ -341,16 +376,133 @@ public final class Layer {
}
/**
- * Closes the registry. Subsequent {@link #register} calls throw. Called by
- * {@code CoreModuleProvider.notifyAfterCompleted()} after every module's prepare/start
- * has run, so MAL/LAL/SPI/yaml all had their full window. Idempotent.
+ * Closes the registry. Subsequent {@link #register} calls throw; only
+ * {@link #registerDynamic} / {@link #unregisterDynamic} can mutate the registry after
+ * seal. Called by {@code CoreModuleProvider.notifyAfterCompleted()} after every
+ * module's prepare/start has run, so MAL/LAL/SPI/yaml all had their full window.
+ * Idempotent.
*/
public static synchronized void seal() {
+ rebuildCachedValues();
+ SEALED = true;
+ }
+
+ /**
+ * Post-seal registration entry point for runtime DSL hot-update. Mirrors
+ * {@link #register} (same name regex, name uniqueness, ordinal uniqueness, idempotent
+ * on identical re-registration) but bypasses the seal check so the runtime-rule module
+ * can introduce layers during a hot-update apply.
+ *
+ * <p>Enforces {@code ordinal >= }{@link #RUNTIME_DYNAMIC_MIN_ORDINAL} so runtime layers
+ * cannot collide with built-in ordinals or boot-time external extensions. The caller
+ * (runtime-rule module's {@code RuntimeLayerRegistry}) is responsible for refcounting
+ * declarations and invoking {@link #unregisterDynamic} when the last declaring rule is
+ * removed.
+ *
+ * <p>Rebuilds the {@link #values()} snapshot so post-seal additions are visible to
+ * subsequent iterators.
+ *
+ * @throws IllegalArgumentException if name shape is invalid or {@code value < }
+ * {@link #RUNTIME_DYNAMIC_MIN_ORDINAL}
+ * @throws IllegalStateException on name/ordinal conflict with an existing layer
+ */
+ public static synchronized Layer registerDynamic(final String name, final int value, final boolean isNormal) {
+ if (value < RUNTIME_DYNAMIC_MIN_ORDINAL) {
+ throw new IllegalArgumentException(
+ "Runtime dynamic layer ordinal must be >= " + RUNTIME_DYNAMIC_MIN_ORDINAL
+ + " (received " + value + " for layer '" + name + "'). "
+ + "Reserved ranges: 0-9999 = built-in Layer constants, "
+ + "10_000-99_999 = layer-extensions.yml + bundled MAL/LAL layerDefinitions.");
+ }
+ if (name == null || !NAME_PATTERN.matcher(name).matches()) {
+ throw new IllegalArgumentException(
+ "Layer name must match [A-Z][A-Z0-9_]*: " + name);
+ }
+ final Layer existingByName = BY_NAME.get(name);
+ if (existingByName != null) {
+ if (existingByName.value == value && existingByName.isNormal == isNormal) {
+ return existingByName;
+ }
+ throw new IllegalStateException(
+ "Layer name conflict: " + name + " already registered as ordinal=" + existingByName.value
+ + ", normal=" + existingByName.isNormal
+ + "; refused re-registration as ordinal=" + value + ", normal=" + isNormal);
+ }
+ final Layer existingByValue = BY_VALUE.get(value);
+ if (existingByValue != null) {
+ throw new IllegalStateException(
+ "Layer ordinal conflict at " + value
+ + ": existing=" + existingByValue.name + ", new=" + name);
+ }
+ final Layer layer = new Layer(name, value, isNormal);
+ BY_VALUE.put(value, layer);
+ BY_NAME.put(name, layer);
+ DYNAMIC_NAMES.add(name);
+ rebuildCachedValues();
+ return layer;
+ }
+
+ /**
+ * Remove a layer added via {@link #registerDynamic}. No-op if the name is not currently
+ * registered. Throws if the named layer was NOT registered through the dynamic
+ * channel — built-in {@code Layer.*} constants, {@code layer-extensions.yml} entries,
+ * and bundled MAL/LAL {@code layerDefinitions:} blocks all funnel through
+ * {@link #register} and are non-removable because their ordinals are baked into
+ * persisted entity primary keys (e.g. {@code ServiceTraffic.id}).
+ *
+ * <p>Ownership is tracked explicitly through {@link #DYNAMIC_NAMES}, not inferred from
+ * the ordinal range — operators are free to put boot-time external layers anywhere in
+ * the registry's accepted range, so an ordinal-only check would wrongly accept removal
+ * of a boot-time layer that happens to live in the {@code >=100_000} tier.
+ *
+ * <p>Rebuilds the {@link #values()} snapshot.
+ *
+ * @throws IllegalStateException if the named layer was not registered via
+ * {@link #registerDynamic}
+ */
+ public static synchronized void unregisterDynamic(final String name) {
+ if (name == null) {
+ return;
+ }
+ final Layer existing = BY_NAME.get(name);
+ if (existing == null) {
+ return;
+ }
+ if (!DYNAMIC_NAMES.contains(name)) {
+ throw new IllegalStateException(
+ "Refusing to unregister non-dynamic layer " + name + " (ordinal="
+ + existing.value + "). Only layers registered through "
+ + "registerDynamic are removable; this one came through register "
+ + "(built-in Layer constant, layer-extensions.yml, or bundled "
+ + "MAL/LAL layerDefinitions).");
+ }
+ BY_NAME.remove(name);
+ BY_VALUE.remove(existing.value);
+ DYNAMIC_NAMES.remove(name);
+ rebuildCachedValues();
+ }
+
+ /**
+ * True when the named layer was registered through {@link #registerDynamic} — i.e.
+ * owned by the runtime-rule module's {@code RuntimeLayerRegistry} rather than by the
+ * boot-time {@link #register} channel. Used by the boot-time MAL/LAL static loaders
+ * to skip {@code layerDefinitions:} entries the runtime path has already handled.
+ */
+ public static boolean isDynamic(final String name) {
+ return name != null && DYNAMIC_NAMES.contains(name);
+ }
+
+ /**
+ * Recompute {@link #CACHED_VALUES} from {@link #BY_VALUE}, sorted by ordinal. Called on
+ * {@link #seal()} and on every post-seal mutation through {@link #registerDynamic} /
+ * {@link #unregisterDynamic}. Pre-seal callers iterating {@link #values()} still hit
+ * the "not yet sealed" guard because {@link #SEALED} flips only inside {@link #seal()}.
+ */
+ private static void rebuildCachedValues() {
CACHED_VALUES = BY_VALUE.values()
.stream()
.sorted(Comparator.comparingInt(Layer::value))
.toArray(Layer[]::new);
- SEALED = true;
}
public static Layer valueOf(final int value) {
@@ -382,6 +534,17 @@ public final class Layer {
return layer;
}
+ /**
+ * Lenient ordinal lookup — returns {@code null} when no layer is registered at the
+ * given ordinal. Intended for callers that need to probe the registry without paying
+ * the cost (or noise) of catching {@link UnexpectedException} thrown by
+ * {@link #valueOf(int)}. Used by the runtime-rule conflict checker when validating
+ * dynamic-layer declarations before any registration is attempted.
+ */
+ public static Layer peekByValue(final int value) {
+ return BY_VALUE.get(value);
+ }
+
/**
* Snapshot of all registered layers, sorted by ordinal value. Returns a fresh array each
* call to preserve enum-style mutation safety; the underlying snapshot is computed once
@@ -394,7 +557,7 @@ public final class Layer {
* or defer the iteration to a post-boot phase.
*/
public static Layer[] values() {
- if (CACHED_VALUES == null) {
+ if (!SEALED) {
throw new IllegalStateException(
"Layer.values() called before the registry was sealed. "
+ "External layers may still register during module prepare()/start(); "
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/rule/ext/RuleSetMerger.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/rule/ext/RuleSetMerger.java
index d22240174c..9478dcee3c 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/rule/ext/RuleSetMerger.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/rule/ext/RuleSetMerger.java
@@ -148,7 +148,14 @@ public final class RuleSetMerger {
resolver.getClass().getName(), catalog, name);
return;
}
- out.put(name, res.getContent());
+ // Only override existing disk entries. Pure-runtime rules (no disk
+ // twin) are applied by RuleSync.runOnce post-seal via the dynamic
+ // channel; injecting them into the static loader here would
+ // register their layerDefinitions through Layer.register and
+ // permanently lose operator-removable ownership.
+ if (out.containsKey(name)) {
+ out.put(name, res.getContent());
+ }
break;
case INACTIVE:
out.remove(name);
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/rule/ext/RuntimeRuleOverrideResolver.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/rule/ext/RuntimeRuleOverrideResolver.java
index c3c3236724..3f0c341123 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/rule/ext/RuntimeRuleOverrideResolver.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/rule/ext/RuntimeRuleOverrideResolver.java
@@ -36,7 +36,11 @@ import org.apache.skywalking.oap.server.library.module.ModuleManager;
* {@code (name -> bytes)} map per catalog, with priority deciding ties:
* <ul>
* <li>Lower {@link #priority} resolvers are applied first; higher priority overwrites.</li>
- * <li>{@link Decision#ACTIVE} substitutes the resolver's content into the merged set.</li>
+ * <li>{@link Decision#ACTIVE} substitutes the resolver's content into the merged set
+ * <strong>only when the key already exists on disk</strong>. Resolver-only entries
+ * (no disk twin) are dropped — they're handled by the runtime-rule reconciler post-seal
+ * through {@code RuntimeLayerRegistry} so layerDefinitions go through the dynamic
+ * channel and remain operator-removable. The merger drops them silently.</li>
* <li>{@link Decision#INACTIVE} removes the entry from the merged set — even if the disk
* file or a lower-priority resolver had content for that key.</li>
* <li>A resolver omits a key from {@link #loadAll} when it has no opinion; the next higher
@@ -48,11 +52,13 @@ import org.apache.skywalking.oap.server.library.module.ModuleManager;
* Resolver A (priority 100, runtime-rule DB):
* "vm" => Resolution(ACTIVE, bytes-from-DB)
* "noisy-rule" => Resolution(INACTIVE, null)
+ * "new-rule" => Resolution(ACTIVE, bytes-from-DB) // no disk twin
*
* Result for catalog "otel-rules":
* - if disk has "vm.yaml": merged["vm"] = bytes-from-DB
* - if disk has "noisy-rule.yaml": merged drops "noisy-rule" entirely
- * - if disk lacks "new-rule.yaml": merged["new-rule"] = bytes-from-DB (DB-only rule)
+ * - if disk lacks "new-rule.yaml": merged does NOT contain "new-rule" (pure runtime —
+ * applied by RuleSync post-seal via the dynamic channel)
* </pre>
*/
public interface RuntimeRuleOverrideResolver {
diff --git a/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/analysis/LayerDynamicRegisterTest.java b/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/analysis/LayerDynamicRegisterTest.java
new file mode 100644
index 0000000000..15790d9b16
--- /dev/null
+++ b/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/analysis/LayerDynamicRegisterTest.java
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.analysis;
+
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Validates {@link Layer#registerDynamic(String, int, boolean)} and
+ * {@link Layer#unregisterDynamic(String)}: the post-seal runtime-DSL registration channel
+ * introduced for hot-update of MAL/LAL rules with new {@code layerDefinitions:} entries.
+ *
+ * <p>Tests assert state via {@link Layer#nameOf(String)} / {@link Layer#valueOf(int)} so the
+ * registry need not be sealed — sealing would taint sibling tests that share the JVM (the
+ * registry is a process-wide singleton). Visibility through {@link Layer#values()} is
+ * covered in the runtime-dynamic-layers L2 integration test, which runs in its own forked
+ * OAP process.
+ *
+ * <p>Ordinals used here live in the runtime tier ({@code >= 100_000}) so collisions with
+ * built-in constants, sibling tests' yaml fixtures (ordinals 1150-1199), and the
+ * conventional boot-time external range (10_000-99_999) are impossible by construction.
+ */
+class LayerDynamicRegisterTest {
+
+ @Test
+ void registerDynamicRejectsOrdinalBelowFloor() {
+ // 99_999 is one below the runtime floor — must be rejected with the operator-actionable
+ // message naming the actual ordinal and reserved ranges.
+ final IllegalArgumentException ex = assertThrows(
+ IllegalArgumentException.class,
+ () -> Layer.registerDynamic("DYN_FLOOR_REJECT", 99_999, true));
+ // Message must surface the ordinal so the operator can correlate with their yaml.
+ assertTrue(ex.getMessage().contains("99999") || ex.getMessage().contains("99_999"),
+ "message must quote the offending ordinal: " + ex.getMessage());
+ assertTrue(ex.getMessage().contains("100000") || ex.getMessage().contains("100_000"),
+ "message must quote the floor: " + ex.getMessage());
+ }
+
+ @Test
+ void registerDynamicAtFloorSucceeds() {
+ final Layer layer = Layer.registerDynamic("DYN_AT_FLOOR", 100_000, true);
+ assertEquals("DYN_AT_FLOOR", layer.name());
+ assertEquals(100_000, layer.value());
+ assertEquals(true, layer.isNormal());
+ // Round-trip lookup.
+ assertSame(layer, Layer.nameOf("DYN_AT_FLOOR"));
+ assertSame(layer, Layer.valueOf(100_000));
+ }
+
+ @Test
+ void registerDynamicIsIdempotentOnIdenticalTriple() {
+ final Layer first = Layer.registerDynamic("DYN_IDEMPOTENT", 100_001, true);
+ final Layer second = Layer.registerDynamic("DYN_IDEMPOTENT", 100_001, true);
+ assertSame(first, second);
+ }
+
+ @Test
+ void registerDynamicRejectsNameConflict() {
+ Layer.registerDynamic("DYN_NAME_CONFLICT", 100_002, true);
+ // Same name, different ordinal.
+ assertThrows(IllegalStateException.class,
+ () -> Layer.registerDynamic("DYN_NAME_CONFLICT", 100_003, true));
+ // Same name, different normal flag.
+ assertThrows(IllegalStateException.class,
+ () -> Layer.registerDynamic("DYN_NAME_CONFLICT", 100_002, false));
+ }
+
+ @Test
+ void registerDynamicRejectsOrdinalCollisionWithDifferentName() {
+ Layer.registerDynamic("DYN_ORDINAL_FIRST", 100_004, true);
+ assertThrows(IllegalStateException.class,
+ () -> Layer.registerDynamic("DYN_ORDINAL_SECOND", 100_004, true));
+ }
+
+ @Test
+ void registerDynamicRejectsBadNameShape() {
+ assertThrows(IllegalArgumentException.class,
+ () -> Layer.registerDynamic("badName", 100_005, true));
+ assertThrows(IllegalArgumentException.class,
+ () -> Layer.registerDynamic("9_LAYER", 100_006, true));
+ assertThrows(IllegalArgumentException.class,
+ () -> Layer.registerDynamic("BAD-NAME", 100_007, true));
+ assertThrows(IllegalArgumentException.class,
+ () -> Layer.registerDynamic(null, 100_008, true));
+ }
+
+ @Test
+ void unregisterDynamicRemovesRegisteredLayer() {
+ Layer.registerDynamic("DYN_REMOVE_ME", 100_009, true);
+ assertNotSame(Layer.UNDEFINED, Layer.nameOf("DYN_REMOVE_ME"));
+
+ Layer.unregisterDynamic("DYN_REMOVE_ME");
+
+ // After removal the lookup falls back to UNDEFINED via nameOf, and valueOf(int) throws.
+ assertSame(Layer.UNDEFINED, Layer.nameOf("DYN_REMOVE_ME"));
+ assertThrows(org.apache.skywalking.oap.server.core.UnexpectedException.class,
+ () -> Layer.valueOf(100_009));
+ }
+
+ @Test
+ void unregisterDynamicIsNoOpOnUnknownName() {
+ // No throw, no side effects observable through nameOf.
+ Layer.unregisterDynamic("DYN_NEVER_REGISTERED");
+ assertSame(Layer.UNDEFINED, Layer.nameOf("DYN_NEVER_REGISTERED"));
+ // Null is also tolerated.
+ Layer.unregisterDynamic(null);
+ }
+
+ @Test
+ void unregisterDynamicRefusesToRemoveBundledLayer() {
+ // MESH is a built-in registered through Layer.register — must NOT be removable
+ // through the runtime path, because its ordinal is baked into persisted
+ // ServiceTraffic primary keys. Ownership is tracked explicitly through
+ // DYNAMIC_NAMES, not inferred from the ordinal range.
+ final IllegalStateException ex = assertThrows(
+ IllegalStateException.class,
+ () -> Layer.unregisterDynamic("MESH"));
+ assertTrue(ex.getMessage().contains("MESH"), ex.getMessage());
+ assertTrue(ex.getMessage().contains("non-dynamic"), ex.getMessage());
+ // The built-in is still in the registry after the failed removal attempt.
+ assertSame(Layer.MESH, Layer.nameOf("MESH"));
+ }
+
+ @Test
+ void isDynamicReflectsRegistrationChannel() {
+ // Layer registered through registerDynamic is marked dynamic.
+ Layer.registerDynamic("DYN_ISDYN_PROBE", 100_020, true);
+ assertTrue(Layer.isDynamic("DYN_ISDYN_PROBE"),
+ "registerDynamic-added layer must report isDynamic=true");
+ // Built-in layer is not dynamic.
+ assertFalse(Layer.isDynamic("MESH"),
+ "built-in MESH must report isDynamic=false");
+ // Unknown name is not dynamic.
+ assertFalse(Layer.isDynamic("__NO_SUCH_LAYER__"));
+ assertFalse(Layer.isDynamic(null));
+ // After unregister the marker is cleared.
+ Layer.unregisterDynamic("DYN_ISDYN_PROBE");
+ assertFalse(Layer.isDynamic("DYN_ISDYN_PROBE"),
+ "unregisterDynamic must clear the dynamic marker");
+ }
+
+ @Test
+ void unregisterDynamicRefusesToRemoveAnOperatorLayerAtRuntimeRange() {
+ // A layer registered through Layer.register (the boot-time channel) at an ordinal
+ // that happens to fall in the runtime range MUST still be refused — the previous
+ // ordinal-only check would have wrongly accepted this case (operator yaml is
+ // recommended to use 10_000-99_999, but the registry's enforced range is
+ // anything-non-colliding). The DYNAMIC_NAMES set is the source of truth.
+ Layer.register("DYN_OPERATOR_AT_RUNTIME_TIER", 100_021, true);
+ final IllegalStateException ex = assertThrows(
+ IllegalStateException.class,
+ () -> Layer.unregisterDynamic("DYN_OPERATOR_AT_RUNTIME_TIER"));
+ assertTrue(ex.getMessage().contains("DYN_OPERATOR_AT_RUNTIME_TIER"), ex.getMessage());
+ assertTrue(ex.getMessage().contains("non-dynamic"), ex.getMessage());
+ // Still in the registry, still resolvable.
+ assertNotSame(Layer.UNDEFINED, Layer.nameOf("DYN_OPERATOR_AT_RUNTIME_TIER"));
+ }
+
+ @Test
+ void registerDynamicAfterUnregisterReusesOrdinal() {
+ // A runtime rule introduces a layer.
+ final Layer first = Layer.registerDynamic("DYN_REUSE_CYCLE", 100_010, true);
+ assertSame(first, Layer.valueOf(100_010));
+
+ // The declaring rule is removed; refcount drops to 0; the layer is freed.
+ Layer.unregisterDynamic("DYN_REUSE_CYCLE");
+
+ // A subsequent registration with the same (name, ordinal, normal) is permitted —
+ // the slot is free.
+ final Layer second = Layer.registerDynamic("DYN_REUSE_CYCLE", 100_010, true);
+ assertNotSame(first, second);
+ assertEquals(100_010, second.value());
+ }
+}
diff --git a/oap-server/server-starter/src/main/resources/layer-extensions.yml b/oap-server/server-starter/src/main/resources/layer-extensions.yml
index b6ade772a9..9214cb94bc 100644
--- a/oap-server/server-starter/src/main/resources/layer-extensions.yml
+++ b/oap-server/server-starter/src/main/resources/layer-extensions.yml
@@ -24,20 +24,35 @@
# every OAP node in a cluster reads and writes the same data, every node MUST agree on
# the (name, ordinal) mapping. Deploy this file identically to every node.
#
-# Recommended ordinal range:
-# 0-999 reserved by convention for the OAP distribution's built-in layers.
-# >= 1000 strongly recommended for external layers. Staying above 1000 avoids
-# colliding with future built-in layers added by the OAP distribution.
-# Conflicting ordinals or names cause OAP boot to fail with a clear error.
+# Ordinal-range convention (enforced for the runtime tier, recommended for the others):
+#
+# 0 – 9_999 built-in Layer.* constants. Currently 0..49 in use; the rest
+# of the range is reserved for future built-ins added by the
+# OAP distribution.
+# 10_000 – 99_999 boot-time external layers: this file (layer-extensions.yml),
+# LayerExtension SPI, and bundled MAL/LAL `layerDefinitions:`
+# blocks. Loaded before the registry is sealed.
+# 100_000 – 2_147_483_647
+# runtime DSL dynamic layers — introduced by MAL/LAL rules
+# pushed through the runtime-rule hot-update API. The ordinal
+# is operator-pinned (same as the other tiers — the OAP does
+# NOT auto-allocate; ordinals end up in persisted
+# ServiceTraffic primary keys and must be operator-stable
+# across restarts). Tracked through `RuntimeLayerRegistry`
+# with refcounted lifetime. NOT for use in this file.
+#
+# Conflicting ordinals or names cause OAP boot to fail with a clear error. Runtime DSL
+# rules that pin `ordinal:` below 100_000 are rejected at apply time with the
+# `layer_ordinal_out_of_range` status in the HTTP response.
#
# Name shape: must match the regex `[A-Z][A-Z0-9_]*` (upper-snake-case, leading letter).
#
# Field reference:
# name: upper-snake-case identifier. MAL/LAL rules and UI templates reference
# the layer by this name.
-# ordinal: stable integer. Once assigned and used to write data, this value is
-# permanent — changing it orphans every previously stored row tagged with
-# the old ordinal.
+# ordinal: stable integer in 10_000-99_999 for entries declared here. Once assigned
+# and used to write data, this value is permanent — changing it orphans
+# every previously stored row tagged with the old ordinal.
# normal: true = services in this layer are agent-installed
# false = services are conjectured/virtual (e.g. inferred from upstream
# calls). Defaults to true if omitted.
@@ -46,18 +61,22 @@
#
# layers:
# - name: IOT_FLEET
-# ordinal: 1000
+# ordinal: 10000
# normal: true
# - name: IOT_GATEWAY
-# ordinal: 1001
+# ordinal: 10001
# normal: true
# - name: VIRTUAL_IOT_DEVICE
-# ordinal: 1002
+# ordinal: 10002
# normal: false
#
# Other places a custom layer can be declared:
-# - At the top of a MAL or LAL rule file via a `layerDefinitions:` block —
-# recommended when a layer ships together with the rules that produce its telemetry.
-# - As an in-process extension contributed by a plugin jar on the OAP classpath.
+# - At the top of a bundled MAL or LAL rule file via a `layerDefinitions:` block —
+# loaded at OAP boot, same 10_000-99_999 ordinal tier as this file.
+# - At the top of a runtime MAL or LAL rule pushed via the hot-update API — operator
+# pins an explicit ordinal in the 100_000+ tier; refcounted across declaring rules;
+# unregistered when the last declaring rule is removed.
+# - As an in-process extension contributed by a plugin jar on the OAP classpath via
+# the LayerExtension SPI (boot-time tier).
layers: []
diff --git a/test/e2e-v2/cases/runtime-rule/mal-storage/banyandb/e2e.yaml b/test/e2e-v2/cases/runtime-rule/mal-storage/banyandb/e2e.yaml
index af1fcededf..fb8033a540 100644
--- a/test/e2e-v2/cases/runtime-rule/mal-storage/banyandb/e2e.yaml
+++ b/test/e2e-v2/cases/runtime-rule/mal-storage/banyandb/e2e.yaml
@@ -22,9 +22,10 @@
# the runtime-rule HTTP handler accepts seed rules that derive SkyWalking
# metrics from those signals; the flow script drives the full lifecycle —
# CREATE → UPDATE-FILTER → UPDATE-STRUCTURAL → DUMP → ILLEGAL × 4 →
-# SHAPE-BREAK → INACTIVATE → ACTIVATE → DELETE → DUMP — and asserts each
-# phase against /list, swctl, the BanyanDB-side measure existence, and the
-# step-label-attributed query results.
+# LAYER-REJECTION × 3 → LAYER-ROUND-TRIP → SHAPE-BREAK → INACTIVATE →
+# ACTIVATE → DELETE → DUMP — and asserts each phase against /list, swctl,
+# the BanyanDB-side measure existence, and the step-label-attributed
+# query results.
setup:
env: compose
diff --git a/test/e2e-v2/cases/runtime-rule/mal-storage/runtime-rule-flow.sh b/test/e2e-v2/cases/runtime-rule/mal-storage/runtime-rule-flow.sh
index 7f80d2b032..97831551aa 100755
--- a/test/e2e-v2/cases/runtime-rule/mal-storage/runtime-rule-flow.sh
+++ b/test/e2e-v2/cases/runtime-rule/mal-storage/runtime-rule-flow.sh
@@ -23,11 +23,15 @@
# 2. UPDATE-FILTER — POST rule v2 (body change ×10, same shape)
# 3. UPDATE-STRUCTURAL — POST rule v3 (adds 2nd metric)
# 4. DUMP (mid-flight) — GET /dump returns tar.gz with the live ruleset
-# 5. ILLEGAL-APPLY × 4 — verify rejection paths
+# 5. ILLEGAL-APPLY — verify rejection paths
# 5a. malformed YAML → 400 compile_failed
# 5b. shape flip without allowStorageChange → 409
# 5c. /delete on ACTIVE row → 409 requires_inactivate_first
# 5d. sibling rule claims the same metric name → 409 ownership conflict
+# 5e. layerDefinitions ordinal <100_000 → 400 layer_ordinal_out_of_range
+# 5f. layerDefinitions name shape invalid → 400 layer_name_invalid
+# 5g. layerDefinitions redeclares built-in MESH → 400 layer_name_conflict
+# 5h. HAPPY-PATH dynamic-LAYER round-trip via swctl `layer ls`
# 6. SHAPE-BREAK — /inactivate → /delete → POST rule v4 (INSTANCE-scope)
# 7. INACTIVATE — POST /inactivate (soft-pause)
# 8. ACTIVATE — re-POST /addOrUpdate (lossless reactivate)
@@ -504,6 +508,113 @@ list_no_row "${SIBLING_NAME}"
|| fail "5d: primary rule's contentHash moved after sibling-conflict rejection"
assert_metric_step_advanced "e2e_rr_requests" "structural" "${struct_baseline}" 180
+# Phase 5e/f/g — dynamic-LAYER rejection paths. Each fixture targets one of the
+# new applyStatus codes (layer_ordinal_out_of_range, layer_name_invalid,
+# layer_name_conflict). Same invariant as 5a-d: response JSON carries the
+# expected applyStatus, /list shows no sibling row, the primary rule's
+# contentHash is unchanged, and structural-bucket aggregation keeps advancing.
+
+log "=== Phase 5e: ILLEGAL layer ordinal below runtime floor ==="
+struct_baseline="$(latest_bucket_id_for_step "e2e_rr_requests" "structural")"
+resp="$(post_rule_expect_status \
+ "${SEED_RULES_DIR}/illegal-layer-out-of-range.yaml" "400" "" "${SIBLING_NAME}")"
+assert_apply_status "layer_ordinal_out_of_range" "${resp}"
+list_no_row "${SIBLING_NAME}"
+[[ "$(list_row ACTIVE | jq -r '.contentHash')" == "${hash_structural}" ]] \
+ || fail "5e: primary rule's contentHash moved after layer-ordinal rejection"
+assert_metric_step_advanced "e2e_rr_requests" "structural" "${struct_baseline}" 180
+
+log "=== Phase 5f: ILLEGAL layer name shape ==="
+struct_baseline="$(latest_bucket_id_for_step "e2e_rr_requests" "structural")"
+resp="$(post_rule_expect_status \
+ "${SEED_RULES_DIR}/illegal-layer-name-invalid.yaml" "400" "" "${SIBLING_NAME}")"
+assert_apply_status "layer_name_invalid" "${resp}"
+list_no_row "${SIBLING_NAME}"
+[[ "$(list_row ACTIVE | jq -r '.contentHash')" == "${hash_structural}" ]] \
+ || fail "5f: primary rule's contentHash moved after layer-name-invalid rejection"
+assert_metric_step_advanced "e2e_rr_requests" "structural" "${struct_baseline}" 180
+
+log "=== Phase 5g: ILLEGAL layer name conflict (built-in MESH redeclared) ==="
+struct_baseline="$(latest_bucket_id_for_step "e2e_rr_requests" "structural")"
+resp="$(post_rule_expect_status \
+ "${SEED_RULES_DIR}/illegal-layer-name-conflict.yaml" "400" "" "${SIBLING_NAME}")"
+assert_apply_status "layer_name_conflict" "${resp}"
+# Message must name the conflicting source so operators see what to align with.
+echo "${resp}" | jq -e '.message | test("built-in")' >/dev/null \
+ || fail "5g: response message did not label source as built-in: ${resp}"
+list_no_row "${SIBLING_NAME}"
+[[ "$(list_row ACTIVE | jq -r '.contentHash')" == "${hash_structural}" ]] \
+ || fail "5g: primary rule's contentHash moved after layer-name-conflict rejection"
+assert_metric_step_advanced "e2e_rr_requests" "structural" "${struct_baseline}" 180
+
+# Phase 5h — HAPPY-PATH dynamic-LAYER round-trip + RESTART survival.
+# POST a sibling pure-runtime rule (no bundled twin) with layerDefinitions,
+# verify swctl layer ls lists it, RESTART the OAP container, verify the
+# layer survives the restart AND remains operator-removable through
+# /inactivate + /delete. Proves the runtime-channel ownership contract
+# end-to-end (the bug RuleSetMerger.merge fix addresses).
+log "=== Phase 5h: HAPPY-PATH + RESTART dynamic-LAYER round-trip ==="
+struct_baseline="$(latest_bucket_id_for_step "e2e_rr_requests" "structural")"
+resp="$(post_rule "${SEED_RULES_DIR}/seed-rule-sibling-with-layer.yaml" "" "${SIBLING_NAME}")"
+assert_apply_status "structural_applied" "${resp}"
+list_row "ACTIVE" "${SIBLING_NAME}" >/dev/null
+sleep 2
+layers_after_create="$(swctl --display yaml \
+ --base-url=http://${OAP_HOST}:${OAP_GQL_PORT}/graphql layer ls)"
+echo "${layers_after_create}" | grep -q "E2E_RR_DYN_LAYER" \
+ || fail "5h-create: layer ls missing E2E_RR_DYN_LAYER (got: ${layers_after_create})"
+log " pre-restart: layer ls includes E2E_RR_DYN_LAYER"
+
+# Restart the OAP container. base-compose.yml pins the OAP image to
+# `skywalking/oap:latest`; filter by that. Do NOT fall back to a bare
+# `grep oap` match — on a shared dev/CI host that can hit unrelated
+# containers (e.g. another OAP, an agent test rig).
+log " restarting OAP container..."
+oap_container="$(docker ps --filter "ancestor=skywalking/oap:latest" \
+ --format '{{.Names}}' | head -1)"
+[ -n "${oap_container}" ] || fail "5h: no skywalking/oap:latest container found — refusing to guess"
+docker restart "${oap_container}" >/dev/null
+# Wait for the REST port to come back. Cap at 180s so a true hang surfaces.
+for i in $(seq 1 90); do
+ if curl -fsS "${REST_BASE}/runtime/rule/list" >/dev/null 2>&1; then
+ log " OAP back up after ${i}*2s"
+ break
+ fi
+ sleep 2
+done
+curl -fsS "${REST_BASE}/runtime/rule/list" >/dev/null \
+ || fail "5h: OAP did not come back online after restart"
+
+# Critical assertion: the runtime layer must still be visible AND retain its
+# dynamic ownership (i.e. removable through /delete below). Without the
+# RuleSetMerger override-only fix, the pure-runtime rule's layerDefinitions
+# would be replayed through Layer.register at boot and the layer would be
+# stuck for the OAP process lifetime.
+sleep 2
+layers_after_restart="$(swctl --display yaml \
+ --base-url=http://${OAP_HOST}:${OAP_GQL_PORT}/graphql layer ls)"
+echo "${layers_after_restart}" | grep -q "E2E_RR_DYN_LAYER" \
+ || fail "5h-restart: layer ls missing E2E_RR_DYN_LAYER after restart (got: ${layers_after_restart})"
+list_row "ACTIVE" "${SIBLING_NAME}" >/dev/null
+log " post-restart: layer + rule survived"
+
+# Now prove the layer is still removable through the dynamic channel.
+retry_curl_post "${REST_BASE}/runtime/rule/inactivate?catalog=${CATALOG}&name=${SIBLING_NAME}" >/dev/null \
+ || fail "5h: sibling inactivate failed"
+retry_curl_post "${REST_BASE}/runtime/rule/delete?catalog=${CATALOG}&name=${SIBLING_NAME}" >/dev/null \
+ || fail "5h: sibling delete failed"
+list_no_row "${SIBLING_NAME}"
+sleep 2
+layers_after_delete="$(swctl --display yaml \
+ --base-url=http://${OAP_HOST}:${OAP_GQL_PORT}/graphql layer ls)"
+echo "${layers_after_delete}" | grep -q "E2E_RR_DYN_LAYER" \
+ && fail "5h-delete: layer ls still includes E2E_RR_DYN_LAYER after sibling delete — runtime ownership lost across restart (RuleSetMerger merger bug?)"
+
+# Primary rule's contentHash unchanged throughout; aggregation resumed after restart.
+[[ "$(list_row ACTIVE | jq -r '.contentHash')" == "${hash_structural}" ]] \
+ || fail "5h: primary rule's contentHash moved during dynamic-layer round-trip"
+assert_metric_step_advanced "e2e_rr_requests" "structural" "${struct_baseline}" 300
+
# Phase 6 — SHAPE-BREAK via the supported route: /inactivate → /delete →
# POST a new shape under the same (catalog, name).
log "=== Phase 6: SHAPE-BREAK ==="
diff --git a/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/illegal-layer-name-conflict.yaml b/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/illegal-layer-name-conflict.yaml
new file mode 100644
index 0000000000..1272618e82
--- /dev/null
+++ b/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/illegal-layer-name-conflict.yaml
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Illegal fixture for Phase 5g: a runtime-DSL layerDefinitions entry
+# redeclaring the built-in MESH layer (ordinal 1) at a runtime ordinal
+# (200_000). The triples differ, so the REST handler must reject with HTTP
+# 400 and applyStatus `layer_name_conflict`, naming the source as
+# "built-in or boot-time extension".
+metricPrefix: e2e_rr_dyn_nc
+layerDefinitions:
+ - name: MESH
+ ordinal: 200000
+ normal: true
+expSuffix: service(['service_name'], Layer.GENERAL)
+metricsRules:
+ - name: nc_test
+ exp: e2e_rr_request_count_total.sum(['service_name'])
diff --git a/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/illegal-layer-name-invalid.yaml b/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/illegal-layer-name-invalid.yaml
new file mode 100644
index 0000000000..8f31030098
--- /dev/null
+++ b/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/illegal-layer-name-invalid.yaml
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Illegal fixture for Phase 5f: a runtime-DSL layerDefinitions entry with a
+# name that contains a hyphen — must match [A-Z][A-Z0-9_]*. The REST handler
+# must reject with HTTP 400 and applyStatus `layer_name_invalid`.
+metricPrefix: e2e_rr_dyn_inv
+layerDefinitions:
+ - name: bad-name
+ ordinal: 200000
+ normal: true
+expSuffix: service(['service_name'], Layer.GENERAL)
+metricsRules:
+ - name: inv_test
+ exp: e2e_rr_request_count_total.sum(['service_name'])
diff --git a/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/illegal-layer-out-of-range.yaml b/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/illegal-layer-out-of-range.yaml
new file mode 100644
index 0000000000..74b1ffb04f
--- /dev/null
+++ b/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/illegal-layer-out-of-range.yaml
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Illegal fixture for Phase 5e: a runtime-DSL layerDefinitions entry with
+# `ordinal: 99999` — the last slot of the boot-time external tier, one slot
+# below the 100_000 runtime floor. This is the most realistic operator
+# confusion: "external layers live in 10_000-99_999, runtime is just more
+# of the same". The runtime-rule REST handler must reject this with HTTP
+# 400 and applyStatus `layer_ordinal_out_of_range`. The metric body is
+# well-formed so the rejection lands on the layer-validation gate, not on
+# a MAL compile error. `normal:` is omitted to confirm the default (true)
+# is honored on the rejection path too.
+metricPrefix: e2e_rr_dyn_ooor
+layerDefinitions:
+ - name: E2E_RR_BAD_RANGE
+ ordinal: 99999
+expSuffix: service(['service_name'], Layer.GENERAL)
+metricsRules:
+ - name: ooor_test
+ exp: e2e_rr_request_count_total.sum(['service_name'])
diff --git a/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/seed-rule-sibling-with-layer.yaml b/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/seed-rule-sibling-with-layer.yaml
new file mode 100644
index 0000000000..58c5f6d331
--- /dev/null
+++ b/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/seed-rule-sibling-with-layer.yaml
@@ -0,0 +1,33 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Happy-path fixture for Phase 5h: a runtime-DSL rule that introduces a NEW
+# layer through `layerDefinitions:` with an explicit `ordinal:` in the
+# runtime tier (>=100_000). `normal:` is intentionally omitted to exercise
+# the LayerDefinition default of `true` (mirrors yaml-extensions + bundled
+# MAL/LAL conventions). The metric body uses the built-in GENERAL layer so
+# this fixture only proves the layer registration, not the cross-layer
+# ingest path (which is covered by the dedicated MAL/L1 unit tests). After
+# /addOrUpdate succeeds, the flow asserts `swctl layer ls` lists
+# E2E_RR_DYN_LAYER; after /delete, the same query proves the layer is
+# unregistered.
+metricPrefix: e2e_rr_dyn_layer
+layerDefinitions:
+ - name: E2E_RR_DYN_LAYER
+ ordinal: 100050
+expSuffix: service(['service_name'], Layer.GENERAL)
+metricsRules:
+ - name: dyn_layer_carrier
+ exp: e2e_rr_request_count_total.sum(['service_name'])