You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by wu...@apache.org on 2026/05/24 09:04:52 UTC
(skywalking) branch feature/runtime-dynamic-layers created (now 2ec4d1cad9)
This is an automated email from the ASF dual-hosted git repository.
wu-sheng pushed a change to branch feature/runtime-dynamic-layers
in repository https://gitbox.apache.org/repos/asf/skywalking.git
at 2ec4d1cad9 Runtime MAL/LAL rules can declare dynamic layers
This branch includes the following new commits:
new 2ec4d1cad9 Runtime MAL/LAL rules can declare dynamic layers
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(skywalking) 01/01: Runtime MAL/LAL rules can declare dynamic layers
Posted by wu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wu-sheng pushed a commit to branch feature/runtime-dynamic-layers
in repository https://gitbox.apache.org/repos/asf/skywalking.git
commit 2ec4d1cad972a937d8e0f12436ff610d0e113434
Author: Wu Sheng <wu...@foxmail.com>
AuthorDate: Sun May 24 17:04:11 2026 +0800
Runtime MAL/LAL rules can declare dynamic layers
Runtime hot-update rules pushed through the runtime-rule admin API may
now declare a layerDefinitions: block. Layers register through a new
Layer.registerDynamic channel (post-seal-allowed) and are refcount-
tracked by RuntimeLayerRegistry: removing the last declaring rule
unregisters the layer. Ordinals are operator-pinned in the 100_000+
tier (no auto-allocation; the ordinal is persisted in ServiceTraffic
primary keys and must be operator-stable across restarts).
Bundled and runtime are strictly separate ownership channels:
* Runtime overrides of bundled rules cannot carry layerDefinitions —
REST rejects with applyStatus=layer_override_forbidden.
* Legacy override rows on disk-twins drop the static-loader
substitution and apply post-seal via the dynamic channel so they
remain operator-removable.
* RuleSetMerger no longer injects resolver-only ACTIVE rows into the
static loader; pure-runtime rules go through RuleSync.runOnce.
Conflict validation surfaces structured 400s with new applyStatus
codes: layer_ordinal_out_of_range, layer_name_conflict,
layer_ordinal_collision, layer_name_invalid, layer_override_forbidden.
The checker covers in-batch duplicates, self-update multi-claimant
ordinal reuse, and bundled-vs-runtime conflicts.
E2E (test/e2e-v2/cases/runtime-rule/mal-storage/runtime-rule-flow.sh)
extended with Phase 5e-5h: three layer-rejection assertions plus a
restart-survival round-trip that POST -> swctl layer ls -> docker
restart -> swctl layer ls still shows -> /delete -> swctl layer ls
no longer shows.
See docs/en/concepts-and-designs/runtime-rule-hot-update.md#dynamic-layers
for the full contract.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
---
CLAUDE.md | 12 +
docs/en/changes/changes.md | 5 +
.../runtime-rule-hot-update.md | 83 +++++
.../oap/log/analyzer/v2/provider/LALConfigs.java | 7 +-
.../provider/LALConfigsLayerDefinitionsTest.java | 1 +
.../meter/analyzer/v2/prometheus/rule/Rules.java | 7 +-
.../prometheus/rule/RulesLayerDefinitionsTest.java | 1 +
.../runtimerule/apply/DeltaClassifier.java | 132 ++++++-
.../receiver/runtimerule/apply/LalFileApplier.java | 114 +++++-
.../receiver/runtimerule/apply/MalFileApplier.java | 108 +++++-
.../runtimerule/engine/lal/LalRuleEngine.java | 14 +
.../runtimerule/engine/mal/MalRuleEngine.java | 15 +
.../extension/DbOverrideRuntimeRuleResolver.java | 75 +++-
.../receiver/runtimerule/layer/AppliedClaims.java | 65 ++++
.../receiver/runtimerule/layer/LayerClaim.java | 74 ++++
.../runtimerule/layer/LayerConflictException.java | 73 ++++
.../layer/RuntimeLayerConflictChecker.java | 290 ++++++++++++++++
.../runtimerule/layer/RuntimeLayerRegistry.java | 385 +++++++++++++++++++++
.../runtimerule/reconcile/DSLRuntimeApply.java | 6 +
.../runtimerule/rest/RuntimeRuleService.java | 66 ++++
.../receiver/runtimerule/state/EngineApplied.java | 17 +
.../runtimerule/apply/MalFileApplierTest.java | 81 +++++
.../DbOverrideRuntimeRuleResolverTest.java | 116 +++++++
.../layer/RuntimeLayerRegistryTest.java | 383 ++++++++++++++++++++
.../skywalking/oap/server/core/analysis/Layer.java | 201 ++++++++++-
.../oap/server/core/rule/ext/RuleSetMerger.java | 9 +-
.../core/rule/ext/RuntimeRuleOverrideResolver.java | 10 +-
.../core/analysis/LayerDynamicRegisterTest.java | 194 +++++++++++
.../src/main/resources/layer-extensions.yml | 47 ++-
.../runtime-rule/mal-storage/banyandb/e2e.yaml | 7 +-
.../runtime-rule/mal-storage/runtime-rule-flow.sh | 113 +++++-
.../seed-rules/illegal-layer-name-conflict.yaml | 29 ++
.../seed-rules/illegal-layer-name-invalid.yaml | 27 ++
.../seed-rules/illegal-layer-out-of-range.yaml | 32 ++
.../seed-rules/seed-rule-sibling-with-layer.yaml | 33 ++
35 files changed, 2748 insertions(+), 84 deletions(-)
diff --git a/CLAUDE.md b/CLAUDE.md
index 1608fda657..7ea2cd5beb 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -155,6 +155,18 @@ public class XxxModuleProvider extends ModuleProvider {
- Use Lombok annotations (`@Getter`, `@Setter`, `@Builder`, `@Data`, `@Slf4j`, etc.)
- Follow existing patterns in similar files
+**Comments:**
+- Don't explain what the code already says. Comment only when reading the code is
+ not enough — invariants that aren't visible in this method, cross-cutting
+ ordering, why a non-obvious branch exists, why a seemingly redundant call is
+ required. If the code is clear, no comment.
+- Connect logic that lives in different places. A check here that exists only
+ because of a guarantee made elsewhere deserves a one-line pointer to that
+ guarantee. A workaround for a known bug elsewhere deserves a link.
+- Trim restated javadoc. Method-level docs should describe the contract, not
+ re-narrate the body. Multi-paragraph rationale belongs in the design doc, not
+ the source.
+
### License Header
Java, XML, and YAML/YML files must include the Apache 2.0 license header (see `HEADER` file).
JSON and Markdown files are excluded (JSON doesn't support comments, see `.licenserc.yaml`).
diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md
index b2804f1359..604b6be4e9 100644
--- a/docs/en/changes/changes.md
+++ b/docs/en/changes/changes.md
@@ -238,6 +238,11 @@
admin-host only" entry above for the public REST retirement.
#### OAP Server
+* Runtime MAL/LAL hot-update rules can declare `layerDefinitions:` to introduce new
+ layers. Ordinals are operator-pinned in the `100_000+` tier; the layer is
+ refcount-tracked and unregistered when the last declaring rule is removed. See
+ [runtime-rule-hot-update.md#dynamic-layers](../concepts-and-designs/runtime-rule-hot-update.md)
+ for the conflict rules and limitations.
* Fix: remove the redundant tags from the `envoy-ai-gateway.yaml` LAL configuration.
* Add Zipkin Virtual GenAI e2e test. Use `zipkin_json` exporter to avoid protobuf dependency conflict
between `opentelemetry-exporter-zipkin-proto-http` (protobuf~=3.12) and `opentelemetry-proto` (protobuf>=5.0).
diff --git a/docs/en/concepts-and-designs/runtime-rule-hot-update.md b/docs/en/concepts-and-designs/runtime-rule-hot-update.md
index 78ee77f6d4..4abf49f7dd 100644
--- a/docs/en/concepts-and-designs/runtime-rule-hot-update.md
+++ b/docs/en/concepts-and-designs/runtime-rule-hot-update.md
@@ -432,6 +432,89 @@ status, per-node `localState`, and `lastApplyError` for any rule whose most rece
apply failed. There is no separate alert channel — `/list` plus the OAP log are
the entire diagnostic surface.
+## Dynamic layers
+
+Runtime MAL/LAL rules MAY introduce new layer names through the same
+`layerDefinitions:` block bundled rule files use:
+
+```yaml
+layerDefinitions:
+ - name: MY_NEW_LAYER
+ ordinal: 100050 # required, runtime tier (>=100_000)
+ # normal: true # optional; defaults to true
+metricPrefix: my_prefix
+metricsRules:
+ - name: requests
+ exp: any.sum(['service', 'instance'])
+```
+
+### Ordinal tier — operator-pinned
+
+| Range | Channel |
+|------------------------------|------------------------------------------------------------------------|
+| `0 – 9_999` | Built-in `Layer.*` constants |
+| `10_000 – 99_999` | `layer-extensions.yml` + bundled MAL/LAL `layerDefinitions:` |
+| `100_000 – Integer.MAX_VALUE`| Runtime DSL dynamic layers |
+
+The OAP does NOT auto-allocate. Ordinals land in persisted `ServiceTraffic` primary
+keys; they must be operator-stable across restarts.
+
+### Lifecycle
+
+`RuntimeLayerRegistry` refcounts each runtime claim per declaring rule. When the last
+runtime claim is removed (`/delete` of the rule, or `/addOrUpdate` that drops the
+entry), the layer is unregistered if it was originally registered through the runtime
+channel.
+
+### Limitations
+
+- **Runtime overrides of bundled rules cannot declare `layerDefinitions:`**. Bundled
+ and runtime are separate layer-ownership channels — never overwrite, never share. The
+ REST handler rejects `/addOrUpdate` of a rule whose name matches a bundled disk file
+ AND whose body carries a non-empty `layerDefinitions:` block with `applyStatus =
+ layer_override_forbidden` (HTTP 400). Operators wanting new layers on a bundled rule
+ must either edit the bundled source on disk and restart, OR push a pure-runtime rule
+ with a different name that declares the layer.
+- **Legacy override rows** (persisted before the rejection was introduced) are handled
+ at boot: the static-loader substitution is dropped so the bundled disk content loads
+ with its original layers, and `RuleSync` then applies the runtime row dynamically
+ post-seal — the runtime row's `layerDefinitions` go through the dynamic channel and
+ are operator-removable via `/inactivate` + `/delete`.
+- **Pure runtime rules (no bundled twin) ARE fully removable**. The layer registers
+ through the runtime channel and `unregisterDynamic` succeeds when the refcount hits
+ zero.
+- **Removing a layer with historical `ServiceTraffic` rows orphans those rows** — the
+ persisted ordinal no longer resolves and reads throw `Unknown Layer value`. Operator
+ must migrate or purge the data before removal; the feature does not auto-clean.
+- **No redeclaration of built-in or boot-time-external layers** at runtime. Use the
+ existing name in your rule body (`layer: GENERAL`, `dest: ...Layer.MESH...`) — do
+ not list those names under `layerDefinitions:`.
+
+### Conflict rules
+
+Validation runs before any apply lands and returns HTTP 400 with the standard
+`{applyStatus, catalog, name, message}` envelope.
+
+| `applyStatus` | Trigger |
+|------------------------------|------------------------------------------------------------------------------------------------------------|
+| `layer_ordinal_out_of_range` | `ordinal:` missing (default 0) or `< 100_000`. |
+| `layer_name_invalid` | `name` does not match `[A-Z][A-Z0-9_]*`. |
+| `layer_name_conflict` | Same `name` already registered with a different `(ordinal, normal)` by another rule (bundled or runtime). Also raised for same-batch duplicate names with different triples, and for self-edit triple changes when another runtime rule shares the layer. |
+| `layer_ordinal_collision` | Different `name`, same `ordinal` already in use. Also raised for same-batch duplicate ordinals, and for self-edit ordinal reuse when another runtime rule still holds the prior layer at that ordinal. |
+| `layer_override_forbidden` | `/addOrUpdate` on a rule with a bundled disk twin where the body contains `layerDefinitions:`. Bundled rules own their layer declarations; runtime overrides may only change the rule body. |
+
+Same triple as an existing bundled layer is permitted as a soft claim (see Limitations
+above). Same triple as an existing runtime layer is a no-op refcount-add. Self-edits
+(changing the triple of a layer this rule already owns) are permitted only when the
+rule is the sole claimant; if other rules share the name, the operator must align both
+sides in lockstep or remove the prior declaration first.
+
+INACTIVE rules hold no layer claims — their refcount entries are dropped at
+`/inactivate`. Reactivation (`/addOrUpdate` against an INACTIVE row) runs full
+validation; if another rule started claiming the name in the meantime,
+`layer_name_conflict` surfaces and the operator must edit or `/delete` the inactive
+rule before it can come back.
+
## What this feature does not do
- **OAL hot-update** is out of scope (see "Scope" above).
diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/provider/LALConfigs.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/provider/LALConfigs.java
index f0dea01585..04f73cf484 100644
--- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/provider/LALConfigs.java
+++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/provider/LALConfigs.java
@@ -71,12 +71,13 @@ public class LALConfigs {
*
* <p>Compared with the legacy disk-only path:
* <ul>
- * <li>Files in {@code files} but missing on disk are still loaded if a resolver
- * contributes ACTIVE content for them (DB-only LAL rules).</li>
* <li>Files on disk + in allow-list with an INACTIVE resolver entry are skipped.</li>
* <li>Files on disk + in allow-list with an ACTIVE resolver entry are parsed from
* resolver bytes (override).</li>
* <li>Files on disk + in allow-list with no resolver opinion are parsed from disk.</li>
+ * <li>Files in resolver contributions but not on disk are NOT loaded here — pure
+ * runtime LAL rules go through {@code RuleSync.runOnce} post-seal via the
+ * dynamic layer channel, which preserves operator-removable ownership.</li>
* </ul>
*/
public static List<LALConfigs> load(final String path, final List<String> files,
@@ -155,7 +156,7 @@ public class LALConfigs {
* Funnel any inline {@code layerDefinitions:} entries through {@code Layer.register}.
* Conflict checks (reserved-range, name uniqueness, ordinal uniqueness, sealed-state) live
* in {@code Layer.register}; failures here surface with the offending rule name in
- * the stack trace, which is enough for an operator to find the bad file.
+ * the stack trace.
*/
private static void registerInlineLayers(final String ruleName, final LALConfigs configs) {
final List<LayerDefinition> defs = configs.getLayerDefinitions();
diff --git a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/provider/LALConfigsLayerDefinitionsTest.java b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/provider/LALConfigsLayerDefinitionsTest.java
index dbf45b6028..5edee31448 100644
--- a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/provider/LALConfigsLayerDefinitionsTest.java
+++ b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/provider/LALConfigsLayerDefinitionsTest.java
@@ -72,4 +72,5 @@ class LALConfigsLayerDefinitionsTest {
assertEquals(1300, Layer.nameOf("TEST_LAL_LAYER_A").value());
}
+
}
diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/prometheus/rule/Rules.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/prometheus/rule/Rules.java
index 5be3b570a9..4b0e369e1e 100644
--- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/prometheus/rule/Rules.java
+++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/prometheus/rule/Rules.java
@@ -131,8 +131,9 @@ public class Rules {
}
// Merge with classpath-discovered resolvers (runtime-rule DB, plus any future
- // priority-ranked source). Resolvers contributing INACTIVE drop their entries;
- // ACTIVE substitutes content. Resolver-only rules (not on disk) are included.
+ // priority-ranked source). ACTIVE substitutes existing disk entries; INACTIVE
+ // drops them. Resolver-only rules (no disk twin) are NOT merged here — they're
+ // applied by RuleSync.runOnce post-seal via the dynamic layer channel.
final Map<String, byte[]> merged = useInstalledManager
? RuleSetMerger.merge(path, diskBytes)
: RuleSetMerger.merge(path, diskBytes, manager);
@@ -161,7 +162,7 @@ public class Rules {
* Funnel any inline {@code layerDefinitions:} entries through {@code Layer.register}.
* Conflict checks (reserved-range, name uniqueness, ordinal uniqueness, sealed-state) live
* in {@code Layer.register}; failures here surface with the offending rule name in
- * the stack trace, which is enough for an operator to find the bad file.
+ * the stack trace.
*/
private static void registerInlineLayers(final String ruleName, final Rule rule) {
final List<LayerDefinition> defs = rule.getLayerDefinitions();
diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/v2/prometheus/rule/RulesLayerDefinitionsTest.java b/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/v2/prometheus/rule/RulesLayerDefinitionsTest.java
index 55811cdf36..0d431845cf 100644
--- a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/v2/prometheus/rule/RulesLayerDefinitionsTest.java
+++ b/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/v2/prometheus/rule/RulesLayerDefinitionsTest.java
@@ -74,4 +74,5 @@ class RulesLayerDefinitionsTest {
assertEquals(1200, Layer.nameOf("TEST_MAL_LAYER_A").value());
}
+
}
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/DeltaClassifier.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/DeltaClassifier.java
index 5548bfba92..6d0e4b42f7 100644
--- a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/DeltaClassifier.java
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/DeltaClassifier.java
@@ -131,19 +131,84 @@ public final class DeltaClassifier {
}
}
- if (added.isEmpty() && removed.isEmpty() && shapeBreak.isEmpty()) {
- // Same metric-name set, same shape for every one. Safe to skip DDL, alarm reset,
- // and L1/L2 drain — the fast path.
- return DSLDelta.filterOnly("body/filter/tag edits only (shapes unchanged)");
+ // layerDefinitions diff escalates to STRUCTURAL even when metric shape is unchanged.
+ // The runtime-layer registry must validate the change atomically with the apply, and
+ // the FILTER_ONLY fast path persists BEFORE apply — a layer conflict surfacing
+ // post-persist would leak a row that subsequently fails to register. STRUCTURAL
+ // runs apply first, so any LayerConflictException surfaces before persist.
+ final String layerDelta = describeMalLayerDelta(oldContent, newContent);
+
+ if (added.isEmpty() && removed.isEmpty() && shapeBreak.isEmpty() && layerDelta == null) {
+ // Same metric-name set, same shape for every one, layerDefinitions unchanged.
+ // Safe to skip DDL, alarm reset, and L1/L2 drain — the fast path.
+ return DSLDelta.filterOnly("body/filter/tag edits only (shapes + layers unchanged)");
}
final Set<String> shapeBreakFrozen = shapeBreak.isEmpty()
? Collections.emptySet()
: Collections.unmodifiableSet(shapeBreak);
- final String reason = reasonFor(added, removed, shapeBreakFrozen);
+ final String reason = layerDelta != null
+ ? reasonFor(added, removed, shapeBreakFrozen) + "; " + layerDelta
+ : reasonFor(added, removed, shapeBreakFrozen);
return DSLDelta.structural(added, removed, shapeBreakFrozen, reason);
}
+ /**
+ * Describe the layer-definitions diff between two MAL YAML strings, or {@code null} when
+ * the lists are identical (or both absent). Both lists are read via the same SnakeYAML
+ * binding the applier uses; a parse failure on either side returns a conservative
+ * "layer parse failed" so the caller escalates rather than risking a partial apply.
+ */
+ private static String describeMalLayerDelta(final String oldContent, final String newContent) {
+ final Set<String> oldLayers = safeLayerSignatures(oldContent);
+ final Set<String> newLayers = safeLayerSignatures(newContent);
+ if (oldLayers.equals(newLayers)) {
+ return null;
+ }
+ final Set<String> added = new LinkedHashSet<>(newLayers);
+ added.removeAll(oldLayers);
+ final Set<String> removed = new LinkedHashSet<>(oldLayers);
+ removed.removeAll(newLayers);
+ if (added.isEmpty() && removed.isEmpty()) {
+ return null;
+ }
+ final StringBuilder sb = new StringBuilder("layerDefinitions changed");
+ if (!added.isEmpty()) {
+ sb.append(" (added=").append(added).append(")");
+ }
+ if (!removed.isEmpty()) {
+ sb.append(" (removed=").append(removed).append(")");
+ }
+ return sb.toString();
+ }
+
+ /** Sticker for a single {@code LayerDefinition} encoding {@code (name, ordinal, normal)}.
+ * Two stickers are equal iff every field matches — i.e., idempotent re-declarations of
+ * the same triple produce identical strings and are correctly treated as no-op. */
+ private static Set<String> safeLayerSignatures(final String content) {
+ if (content == null || content.isEmpty()) {
+ return Collections.emptySet();
+ }
+ try (StringReader reader = new StringReader(content)) {
+ final Rule rule = new Yaml().loadAs(reader, Rule.class);
+ if (rule == null || rule.getLayerDefinitions() == null
+ || rule.getLayerDefinitions().isEmpty()) {
+ return Collections.emptySet();
+ }
+ final Set<String> out = new LinkedHashSet<>();
+ for (final org.apache.skywalking.oap.server.core.analysis.LayerDefinition d
+ : rule.getLayerDefinitions()) {
+ out.add(d.getName() + ":" + d.getOrdinal() + ":" + d.isNormal());
+ }
+ return out;
+ } catch (final RuntimeException ex) {
+ // Conservative on parse failure: caller treats non-null as a change, so reporting
+ // an empty set when one side fails and a non-empty otherwise still escalates.
+ // Return a sentinel so equals() reports a diff regardless of the other side.
+ return Collections.singleton("__layer_parse_failed__:" + ex.getMessage());
+ }
+ }
+
/**
* Classify a LAL rule-file delta. LAL has no direct metric-name target for alarm windows
* (rule keys are {@code (layer, ruleName)} pairs, not metric names), so the added/removed/
@@ -170,9 +235,62 @@ public final class DeltaClassifier {
// prior windows existed).
return DSLDelta.newRule(Collections.emptySet());
}
+ // Layer-definition diffs are surfaced in the reason string so operators can see why
+ // a FILTER_ONLY-shaped edit was classified STRUCTURAL. LAL currently classifies all
+ // content changes as STRUCTURAL anyway (no per-rule shape extraction is implemented),
+ // so the layer diff just enriches the reason rather than changing the verdict.
+ final String layerDelta = describeLalLayerDelta(oldContent, newContent);
+ final String reason = layerDelta != null
+ ? "LAL content changed; " + layerDelta
+ : "LAL content changed";
return DSLDelta.structural(
- Collections.emptySet(), Collections.emptySet(), Collections.emptySet(),
- "LAL content changed");
+ Collections.emptySet(), Collections.emptySet(), Collections.emptySet(), reason);
+ }
+
+ /** LAL variant of {@link #describeMalLayerDelta} — reads from {@link LALConfigs}
+ * instead of {@link Rule}. Identical semantics. */
+ private static String describeLalLayerDelta(final String oldContent, final String newContent) {
+ final Set<String> oldLayers = safeLalLayerSignatures(oldContent);
+ final Set<String> newLayers = safeLalLayerSignatures(newContent);
+ if (oldLayers.equals(newLayers)) {
+ return null;
+ }
+ final Set<String> added = new LinkedHashSet<>(newLayers);
+ added.removeAll(oldLayers);
+ final Set<String> removed = new LinkedHashSet<>(oldLayers);
+ removed.removeAll(newLayers);
+ if (added.isEmpty() && removed.isEmpty()) {
+ return null;
+ }
+ final StringBuilder sb = new StringBuilder("layerDefinitions changed");
+ if (!added.isEmpty()) {
+ sb.append(" (added=").append(added).append(")");
+ }
+ if (!removed.isEmpty()) {
+ sb.append(" (removed=").append(removed).append(")");
+ }
+ return sb.toString();
+ }
+
+ private static Set<String> safeLalLayerSignatures(final String content) {
+ if (content == null || content.isEmpty()) {
+ return Collections.emptySet();
+ }
+ try (StringReader reader = new StringReader(content)) {
+ final LALConfigs configs = new Yaml().loadAs(reader, LALConfigs.class);
+ if (configs == null || configs.getLayerDefinitions() == null
+ || configs.getLayerDefinitions().isEmpty()) {
+ return Collections.emptySet();
+ }
+ final Set<String> out = new LinkedHashSet<>();
+ for (final org.apache.skywalking.oap.server.core.analysis.LayerDefinition d
+ : configs.getLayerDefinitions()) {
+ out.add(d.getName() + ":" + d.getOrdinal() + ":" + d.isNormal());
+ }
+ return out;
+ } catch (final RuntimeException ex) {
+ return Collections.singleton("__layer_parse_failed__:" + ex.getMessage());
+ }
}
/**
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/LalFileApplier.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/LalFileApplier.java
index cdca5ca236..415e81172a 100644
--- a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/LalFileApplier.java
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/LalFileApplier.java
@@ -31,12 +31,16 @@ import lombok.extern.slf4j.Slf4j;
import org.apache.skywalking.oap.log.analyzer.v2.module.LogAnalyzerModule;
import org.apache.skywalking.oap.log.analyzer.v2.provider.LALConfig;
import org.apache.skywalking.oap.server.core.analysis.Layer;
+import org.apache.skywalking.oap.server.core.analysis.LayerDefinition;
import org.apache.skywalking.oap.log.analyzer.v2.provider.LALConfigs;
import org.apache.skywalking.oap.log.analyzer.v2.provider.log.listener.LogFilterListener;
import org.apache.skywalking.oap.server.core.classloader.Catalog;
import org.apache.skywalking.oap.server.core.classloader.DSLClassLoaderManager;
import org.apache.skywalking.oap.server.core.classloader.RuleClassLoader;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.AppliedClaims;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.LayerClaim;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.RuntimeLayerRegistry;
import org.apache.skywalking.oap.server.receiver.runtimerule.state.EngineApplied;
import org.yaml.snakeyaml.Yaml;
@@ -54,9 +58,17 @@ import org.yaml.snakeyaml.Yaml;
public class LalFileApplier {
private final LogFilterListener.Factory factory;
+ private final RuntimeLayerRegistry layerRegistry;
public LalFileApplier(final LogFilterListener.Factory factory) {
+ this(factory, RuntimeLayerRegistry.INSTANCE);
+ }
+
+ /** Test-friendly constructor — pass a dedicated registry to isolate from JVM-wide state. */
+ public LalFileApplier(final LogFilterListener.Factory factory,
+ final RuntimeLayerRegistry layerRegistry) {
this.factory = factory;
+ this.layerRegistry = layerRegistry;
}
/**
@@ -69,9 +81,10 @@ public class LalFileApplier {
* later inside {@link #apply}).
*/
public List<RegisteredRule> planKeys(final String yamlContent, final String sourceName) throws ApplyException {
- final List<LALConfig> configs = parse(yamlContent, sourceName);
- final List<RegisteredRule> keys = new ArrayList<>(configs.size());
- for (final LALConfig c : configs) {
+ final LALConfigs configs = parse(yamlContent, sourceName);
+ final List<LALConfig> rules = configs.getRules();
+ final List<RegisteredRule> keys = new ArrayList<>(rules.size());
+ for (final LALConfig c : rules) {
final boolean isAuto = LALConfig.LAYER_AUTO.equalsIgnoreCase(c.getLayer());
final Layer layer = isAuto
? null
@@ -116,7 +129,18 @@ public class LalFileApplier {
public Applied apply(final String yamlContent, final String sourceName,
final String contentHash,
final DSLClassLoaderManager.Kind kind) throws ApplyException {
- final List<LALConfig> configs = parse(yamlContent, sourceName);
+ final LALConfigs configs = parse(yamlContent, sourceName);
+ final List<LALConfig> rules = configs.getRules();
+ final List<LayerClaim> layerClaims = extractLayerClaims(configs);
+
+ // Register any declared layers FIRST so the LAL `layer:` field on rules in this file
+ // can resolve against the freshly-introduced names before Phase 1 compile starts.
+ // Conflicts throw LayerConflictException (REST handler translates to HTTP 400 with
+ // the structured envelope); atomic — validate() rejects before any registration.
+ final String catalogStr = deriveCatalog(sourceName);
+ final String ruleNameStr = deriveRuleName(sourceName);
+ final AppliedClaims appliedClaims = layerRegistry.apply(
+ RuntimeLayerRegistry.ruleId(catalogStr, ruleNameStr), layerClaims);
// One per-file RuleClassLoader for the whole file — every rule inside shares it, so all
// generated LalExpression classes (one per rule) drop together on unregister when the
@@ -148,13 +172,16 @@ public class LalFileApplier {
// volatile map write; a partial-failure window here is theoretical (Map.put
// doesn't throw). We still track progress per-rule so if somehow the JVM throws
// during phase 2, the caller's rollback list is accurate.
- final List<LogFilterListener.Factory.CompiledLAL> compiled = new ArrayList<>(configs.size());
- for (final LALConfig c : configs) {
+ final List<LogFilterListener.Factory.CompiledLAL> compiled = new ArrayList<>(rules.size());
+ for (final LALConfig c : rules) {
c.setSourceName(sourceName);
try {
compiled.add(factory.compile(c, pool, ruleLoader));
} catch (final Throwable t) {
// Compile-phase failure: zero registrations landed, so partial is empty.
+ // Roll back the layer registrations done above so a failed compile does not
+ // leak runtime-layer state.
+ layerRegistry.rollback(appliedClaims);
throw new ApplyException(
"LAL compile failed for rule '" + c.getName() + "' in " + sourceName,
t, Collections.emptyList());
@@ -172,12 +199,51 @@ public class LalFileApplier {
factory.addOrReplace(x);
registered.add(new RegisteredRule(x.layer, x.ruleName));
} catch (final Throwable t) {
+ // Roll back registrations made so far AND the layer claims. Rule-registration
+ // partial state survives in the caller's `partial` list for unwinding.
+ layerRegistry.rollback(appliedClaims);
throw new ApplyException(
"LAL register failed for rule '" + x.ruleName + "' in " + sourceName,
t, Collections.unmodifiableList(new ArrayList<>(registered)));
}
}
- return new Applied(sourceName, Collections.unmodifiableList(registered), ruleLoader);
+ return new Applied(sourceName, Collections.unmodifiableList(registered), ruleLoader,
+ appliedClaims);
+ }
+
+ /** Split {@code "catalog/name"} → catalog half. Falls back to {@code lal} for bare
+ * source names (legacy / test callers). */
+ private static String deriveCatalog(final String sourceName) {
+ final int idx = sourceName.indexOf('/');
+ return idx > 0 ? sourceName.substring(0, idx) : "lal";
+ }
+
+ private static String deriveRuleName(final String sourceName) {
+ final int idx = sourceName.indexOf('/');
+ return idx > 0 ? sourceName.substring(idx + 1) : sourceName;
+ }
+
+ /**
+ * Build {@link LayerClaim}s for every entry in {@code configs.layerDefinitions}.
+ * Empty input returns an empty list — callers may pass it straight to
+ * {@link RuntimeLayerRegistry#apply}, which is a no-op for an empty list.
+ *
+ * <p>{@code ordinal:} is mandatory in runtime DSL — the registry-side validation
+ * rejects any entry whose ordinal is below {@link Layer#RUNTIME_DYNAMIC_MIN_ORDINAL}
+ * (including the default {@code 0} that an omitted-yaml-key produces) with applyStatus
+ * {@code layer_ordinal_out_of_range}. {@code normal:} is optional and defaults to
+ * {@code true}.
+ */
+ private static List<LayerClaim> extractLayerClaims(final LALConfigs configs) {
+ final List<LayerDefinition> defs = configs.getLayerDefinitions();
+ if (defs == null || defs.isEmpty()) {
+ return Collections.emptyList();
+ }
+ final List<LayerClaim> out = new ArrayList<>(defs.size());
+ for (final LayerDefinition def : defs) {
+ out.add(new LayerClaim(def.getName(), def.getOrdinal(), def.isNormal()));
+ }
+ return out;
}
/**
@@ -249,7 +315,7 @@ public class LalFileApplier {
}
}
- private List<LALConfig> parse(final String yamlContent, final String sourceName) throws ApplyException {
+ private LALConfigs parse(final String yamlContent, final String sourceName) throws ApplyException {
try (StringReader reader = new StringReader(yamlContent)) {
final LALConfigs configs = new Yaml().loadAs(reader, LALConfigs.class);
if (configs == null || configs.getRules() == null || configs.getRules().isEmpty()) {
@@ -257,16 +323,10 @@ public class LalFileApplier {
"LAL YAML parsed to empty/malformed — no rules list in " + sourceName,
null, Collections.emptyList());
}
- if (configs.getLayerDefinitions() != null && !configs.getLayerDefinitions().isEmpty()) {
- throw new ApplyException(
- "LAL rule " + sourceName + " declares `layerDefinitions:`, which is only "
- + "permitted in bundled rule files loaded at OAP boot. Layer extensions "
- + "cannot be added at runtime — the registry is sealed once OAP startup "
- + "completes. Declare the layer in `layer-extensions.yml` or a bundled "
- + "MAL/LAL file, restart OAP, then retry this update.",
- null, Collections.emptyList());
- }
- return configs.getRules();
+ // layerDefinitions: are now permitted in runtime LAL rules; the apply path
+ // funnels them through the runtime-layer registry. The rejection that used to
+ // live here was removed when runtime dynamic layers became a first-class feature.
+ return configs;
} catch (final ApplyException e) {
throw e;
} catch (final Throwable t) {
@@ -290,16 +350,27 @@ public class LalFileApplier {
*/
@Getter
private final RuleClassLoader ruleClassLoader;
+ /** {@code null} when this LAL file declared no {@code layerDefinitions:}; otherwise
+ * the rollback token for the runtime-layer registry mutation this apply effected. */
+ @Getter
+ private final AppliedClaims appliedLayerClaims;
public Applied(final String sourceName, final List<RegisteredRule> registered) {
- this(sourceName, registered, null);
+ this(sourceName, registered, null, null);
}
public Applied(final String sourceName, final List<RegisteredRule> registered,
final RuleClassLoader ruleClassLoader) {
+ this(sourceName, registered, ruleClassLoader, null);
+ }
+
+ public Applied(final String sourceName, final List<RegisteredRule> registered,
+ final RuleClassLoader ruleClassLoader,
+ final AppliedClaims appliedLayerClaims) {
this.sourceName = sourceName;
this.registered = registered;
this.ruleClassLoader = ruleClassLoader;
+ this.appliedLayerClaims = appliedLayerClaims;
}
@Override
@@ -365,6 +436,11 @@ public class LalFileApplier {
return Collections.emptySet();
}
+ @Override
+ public AppliedClaims appliedLayerClaims() {
+ return appliedLayerClaims;
+ }
+
private List<String> ruleKeys() {
final List<String> keys = new ArrayList<>(registered.size());
for (final RegisteredRule r : registered) {
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/MalFileApplier.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/MalFileApplier.java
index fb015f6f45..25d1c349a7 100644
--- a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/MalFileApplier.java
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/MalFileApplier.java
@@ -19,9 +19,11 @@
package org.apache.skywalking.oap.server.receiver.runtimerule.apply;
import java.io.StringReader;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
+import java.util.List;
import java.util.Map;
import java.util.Set;
import javassist.ClassPool;
@@ -32,12 +34,16 @@ import org.apache.skywalking.oap.meter.analyzer.v2.MetricConvert;
import org.apache.skywalking.oap.meter.analyzer.v2.prometheus.rule.MetricsRule;
import org.apache.skywalking.oap.meter.analyzer.v2.prometheus.rule.Rule;
import org.apache.skywalking.oap.server.core.CoreModule;
+import org.apache.skywalking.oap.server.core.analysis.LayerDefinition;
import org.apache.skywalking.oap.server.core.analysis.meter.MeterSystem;
import org.apache.skywalking.oap.server.core.classloader.Catalog;
import org.apache.skywalking.oap.server.core.classloader.DSLClassLoaderManager;
import org.apache.skywalking.oap.server.core.classloader.RuleClassLoader;
import org.apache.skywalking.oap.server.core.storage.model.StorageManipulationOpt;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.AppliedClaims;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.LayerClaim;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.RuntimeLayerRegistry;
import org.apache.skywalking.oap.server.receiver.runtimerule.state.EngineApplied;
import org.yaml.snakeyaml.Yaml;
@@ -63,9 +69,18 @@ import org.yaml.snakeyaml.Yaml;
public class MalFileApplier {
private final MeterSystem meterSystem;
+ private final RuntimeLayerRegistry layerRegistry;
public MalFileApplier(final MeterSystem meterSystem) {
+ this(meterSystem, RuntimeLayerRegistry.INSTANCE);
+ }
+
+ /** Test-friendly constructor: injects a dedicated layer registry so unit / integration
+ * tests don't share JVM-wide state with sibling tests. Production callers go through
+ * the singleton-binding constructor above. */
+ public MalFileApplier(final MeterSystem meterSystem, final RuntimeLayerRegistry layerRegistry) {
this.meterSystem = meterSystem;
+ this.layerRegistry = layerRegistry;
}
/**
@@ -100,6 +115,18 @@ public class MalFileApplier {
final DSLClassLoaderManager.Kind kind) throws ApplyException {
final Rule rule = parse(yamlContent, sourceName);
final Set<String> metricNames = enumerateMetricNames(rule);
+ final List<LayerClaim> layerClaims = extractLayerClaims(rule);
+
+ // Register any declared layers FIRST. Conflicts throw LayerConflictException
+ // (caller translates to HTTP 400 with the structured envelope); per-apply atomic
+ // because validate() runs before any registration lands. After this call, the
+ // Layer registry knows the new layer names; MeterSystem registration below can
+ // safely reference them. On any failure further down we roll back through
+ // RuntimeLayerRegistry.rollback(appliedClaims).
+ final AppliedClaims appliedClaims =
+ layerRegistry.apply(RuntimeLayerRegistry.ruleId(deriveCatalog(sourceName),
+ deriveRuleName(sourceName)),
+ layerClaims);
// Per-file RuleClassLoader + Javassist ClassPool. The pool is parented to the default
// pool so shipped ancestor classes (SumFunction, HistogramFunction, ...) resolve via
@@ -127,7 +154,9 @@ public class MalFileApplier {
// MeterSystem — the caller uses this set for rollback. Passing the full enumerated
// set here would remove metrics the old bundle still owns (disastrous on
// FILTER_ONLY edits, where by definition every metric name is also in the old
- // bundle).
+ // bundle). The layer-registry changes are reverted now so a failed MeterSystem
+ // register does not leak a half-applied layer state.
+ layerRegistry.rollback(appliedClaims);
throw new ApplyException(
"MAL register failed for " + sourceName + " (partial)",
pre.getCause() == null ? pre : pre.getCause(),
@@ -136,9 +165,22 @@ public class MalFileApplier {
// Phase-1 compile failure or other pre-register throw. Nothing was registered with
// MeterSystem, so rollback set is empty — passing a non-empty set would cause the
// caller to unregister metrics the old bundle owns and this apply never touched.
+ layerRegistry.rollback(appliedClaims);
throw new ApplyException("MAL compile failed for " + sourceName, t, Collections.emptySet());
}
- return new Applied(rule, convert, metricNames, ruleLoader);
+ return new Applied(rule, convert, metricNames, ruleLoader, appliedClaims);
+ }
+
+ /** Split {@code "catalog/name"} → catalog half. Falls back to {@code otel-rules} when the
+ * source name is bare (legacy callers, tests). */
+ private static String deriveCatalog(final String sourceName) {
+ final int idx = sourceName.indexOf('/');
+ return idx > 0 ? sourceName.substring(0, idx) : "otel-rules";
+ }
+
+ private static String deriveRuleName(final String sourceName) {
+ final int idx = sourceName.indexOf('/');
+ return idx > 0 ? sourceName.substring(idx + 1) : sourceName;
}
/**
@@ -217,15 +259,9 @@ public class MalFileApplier {
if (rule.getName() == null || rule.getName().isEmpty()) {
rule.setName(sourceName);
}
- if (rule.getLayerDefinitions() != null && !rule.getLayerDefinitions().isEmpty()) {
- throw new ApplyException(
- "MAL rule " + sourceName + " declares `layerDefinitions:`, which is only "
- + "permitted in bundled rule files loaded at OAP boot. Layer extensions "
- + "cannot be added at runtime — the registry is sealed once OAP startup "
- + "completes. Declare the layer in `layer-extensions.yml` or a bundled "
- + "MAL/LAL file, restart OAP, then retry this update.",
- null, Collections.emptySet());
- }
+ // layerDefinitions: are now permitted in runtime MAL rules; they're handled by
+ // the layer registry on the apply path below. The rejection that used to live
+ // here was removed when runtime dynamic layers became a first-class feature.
return rule;
} catch (final ApplyException e) {
throw e;
@@ -234,6 +270,32 @@ public class MalFileApplier {
}
}
+ /**
+ * Build {@link LayerClaim}s for every {@code layerDefinitions:} entry in {@code rule}.
+ * The list is ordered the same as the YAML for stable diagnostic output.
+ *
+ * <p>{@code ordinal:} is mandatory in runtime DSL — the registry-side validation rejects
+ * any entry whose ordinal is below {@link Layer#RUNTIME_DYNAMIC_MIN_ORDINAL} (including
+ * the default {@code 0} that an omitted-yaml-key produces) with applyStatus
+ * {@code layer_ordinal_out_of_range}. {@code normal:} is optional and defaults to
+ * {@code true} via {@link LayerDefinition}'s field initializer — same semantics as the
+ * bundled {@code layerDefinitions:} blocks and {@code layer-extensions.yml}.
+ *
+ * <p>An empty {@code layerDefinitions:} block returns an empty list — callers may pass
+ * it straight to {@link RuntimeLayerRegistry#apply}, which is a no-op for empty input.
+ */
+ private static List<LayerClaim> extractLayerClaims(final Rule rule) {
+ final List<LayerDefinition> defs = rule.getLayerDefinitions();
+ if (defs == null || defs.isEmpty()) {
+ return Collections.emptyList();
+ }
+ final List<LayerClaim> out = new ArrayList<>(defs.size());
+ for (final LayerDefinition def : defs) {
+ out.add(new LayerClaim(def.getName(), def.getOrdinal(), def.isNormal()));
+ }
+ return out;
+ }
+
/**
* Parse raw YAML and return the set of metric names the rule file would register.
* Static-only variant — does not compile, does not register, does not construct a
@@ -299,14 +361,33 @@ public class MalFileApplier {
*/
@Getter
private final RuleClassLoader ruleClassLoader;
+ /**
+ * Snapshot of the {@link RuntimeLayerRegistry} mutation this apply effected. {@code
+ * null} for synthetic {@link Applied}s built by {@code recordBundledClaims} (boot-
+ * time static-rule shadow), which never touch runtime layers. The orchestrator's
+ * commit / discard tail calls {@link RuntimeLayerRegistry#rollback} with this
+ * token on external failure (persist, peer-rejected commit); inside
+ * {@link MalFileApplier#apply} a thrown {@code MetricConvert} construction already
+ * rolls back before re-throwing.
+ */
+ @Getter
+ private final AppliedClaims appliedLayerClaims;
public Applied(final Rule rule, final MetricConvert metricConvert,
final Set<String> registeredMetricNames,
final RuleClassLoader ruleClassLoader) {
+ this(rule, metricConvert, registeredMetricNames, ruleClassLoader, null);
+ }
+
+ public Applied(final Rule rule, final MetricConvert metricConvert,
+ final Set<String> registeredMetricNames,
+ final RuleClassLoader ruleClassLoader,
+ final AppliedClaims appliedLayerClaims) {
this.rule = rule;
this.metricConvert = metricConvert;
this.registeredMetricNames = registeredMetricNames;
this.ruleClassLoader = ruleClassLoader;
+ this.appliedLayerClaims = appliedLayerClaims;
}
@Override
@@ -362,6 +443,11 @@ public class MalFileApplier {
? Collections.emptySet()
: registeredMetricNames;
}
+
+ @Override
+ public AppliedClaims appliedLayerClaims() {
+ return appliedLayerClaims;
+ }
}
/**
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/engine/lal/LalRuleEngine.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/engine/lal/LalRuleEngine.java
index 54ebcc61c2..4bb08ea1ac 100644
--- a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/engine/lal/LalRuleEngine.java
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/engine/lal/LalRuleEngine.java
@@ -43,6 +43,8 @@ import org.apache.skywalking.oap.server.receiver.runtimerule.state.EngineApplied
import org.apache.skywalking.oap.server.library.module.ModuleManager;
import org.apache.skywalking.oap.server.receiver.runtimerule.apply.DeltaClassifier;
import org.apache.skywalking.oap.server.receiver.runtimerule.apply.LalFileApplier;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.AppliedClaims;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.RuntimeLayerRegistry;
import org.apache.skywalking.oap.server.receiver.runtimerule.engine.ApplyInputs;
import org.apache.skywalking.oap.server.receiver.runtimerule.engine.Classification;
import org.apache.skywalking.oap.server.receiver.runtimerule.engine.CompiledDSL;
@@ -413,6 +415,14 @@ public final class LalRuleEngine implements RuleEngine<LalApplyContext> {
@Override
public void rollback(final CompiledDSL compiled, final LalApplyContext ctx) {
final CompiledLalDSL c = (CompiledLalDSL) compiled;
+ // Drop layer claims first so a failed factory rollback below does not strand
+ // runtime-layer state. Safe ordering: layer registry mutations are independent
+ // of the LogFilter factory's rule-key map.
+ final AppliedClaims layerClaims = c.getNewApplied() == null
+ ? null : c.getNewApplied().appliedLayerClaims();
+ if (layerClaims != null && !layerClaims.isNoOp()) {
+ RuntimeLayerRegistry.INSTANCE.rollback(layerClaims);
+ }
if (c.getNewApplied() == null || c.getNewApplied().getRegistered().isEmpty()) {
return;
}
@@ -490,6 +500,10 @@ public final class LalRuleEngine implements RuleEngine<LalApplyContext> {
public void unregister(final String catalog, final String name, final LalApplyContext ctx) {
final String key = DSLScriptKey.key(catalog, name);
final String sourceName = catalog + "/" + name;
+ // Drop every layer this rule claimed. Net-zero layers cascade to
+ // Layer.unregisterDynamic. Independent of the factory teardown below.
+ RuntimeLayerRegistry.INSTANCE.removeRule(
+ RuntimeLayerRegistry.ruleId(catalog, name));
final LalFileApplier.Applied priorLal = appliedFor(ctx.getRules(), key);
if (priorLal != null) {
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/engine/mal/MalRuleEngine.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/engine/mal/MalRuleEngine.java
index 0d6032db66..7da4c32dbb 100644
--- a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/engine/mal/MalRuleEngine.java
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/engine/mal/MalRuleEngine.java
@@ -51,6 +51,8 @@ import org.apache.skywalking.oap.server.core.storage.model.StorageManipulationOp
import org.apache.skywalking.oap.server.receiver.runtimerule.apply.DSLDelta;
import org.apache.skywalking.oap.server.receiver.runtimerule.apply.DeltaClassifier;
import org.apache.skywalking.oap.server.receiver.runtimerule.apply.MalFileApplier;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.AppliedClaims;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.RuntimeLayerRegistry;
import org.apache.skywalking.oap.server.receiver.runtimerule.engine.ApplyInputs;
import org.apache.skywalking.oap.server.receiver.runtimerule.engine.Classification;
import org.apache.skywalking.oap.server.receiver.runtimerule.engine.CompiledDSL;
@@ -606,6 +608,14 @@ public final class MalRuleEngine implements RuleEngine<MalApplyContext> {
@Override
public void rollback(final CompiledDSL compiled, final MalApplyContext ctx) {
final CompiledMalDSL c = (CompiledMalDSL) compiled;
+ // Drop layer claims FIRST so a failed registry rollback doesn't strand metrics
+ // pointing at layers we don't know about. The registry's rollback is bounded —
+ // worst case it logs and proceeds — so this ordering is safe.
+ final AppliedClaims layerClaims = c.getNewApplied() == null
+ ? null : c.getNewApplied().appliedLayerClaims();
+ if (layerClaims != null && !layerClaims.isNoOp()) {
+ RuntimeLayerRegistry.INSTANCE.rollback(layerClaims);
+ }
if (c.getAddedPlusShapeBreak().isEmpty()) {
return;
}
@@ -658,6 +668,11 @@ public final class MalRuleEngine implements RuleEngine<MalApplyContext> {
public void unregister(final String catalog, final String name, final MalApplyContext ctx) {
final String key = DSLScriptKey.key(catalog, name);
final String sourceName = catalog + "/" + name;
+ // Drop every layer this rule claimed. Net-zero layers cascade through
+ // RuntimeLayerRegistry.removeRule → Layer.unregisterDynamic. Safe to call before
+ // the metric teardown below — layers and metrics are independent registries.
+ RuntimeLayerRegistry.INSTANCE.removeRule(
+ RuntimeLayerRegistry.ruleId(catalog, name));
// Always drop the MalConverterRegistry entry for this (catalog, name). The key
// namespace is shared between boot-time and runtime converters, so this single call
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/extension/DbOverrideRuntimeRuleResolver.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/extension/DbOverrideRuntimeRuleResolver.java
index 0e5eb6b3e5..a8e4645c61 100644
--- a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/extension/DbOverrideRuntimeRuleResolver.java
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/extension/DbOverrideRuntimeRuleResolver.java
@@ -19,16 +19,22 @@
package org.apache.skywalking.oap.server.receiver.runtimerule.extension;
import java.io.IOException;
+import java.io.StringReader;
import java.nio.charset.StandardCharsets;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.log.analyzer.v2.provider.LALConfigs;
+import org.apache.skywalking.oap.meter.analyzer.v2.prometheus.rule.Rule;
+import org.apache.skywalking.oap.server.core.analysis.LayerDefinition;
import org.apache.skywalking.oap.server.core.rule.ext.RuntimeRuleOverrideResolver;
+import org.apache.skywalking.oap.server.core.rule.ext.StaticRuleRegistry;
import org.apache.skywalking.oap.server.core.storage.StorageModule;
import org.apache.skywalking.oap.server.core.storage.management.RuntimeRuleManagementDAO;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.yaml.snakeyaml.Yaml;
/**
* Serves operator-supplied rule overrides (rows in the {@code runtime_rule} management table)
@@ -145,14 +151,81 @@ public final class DbOverrideRuntimeRuleResolver implements RuntimeRuleOverrideR
if ("INACTIVE".equalsIgnoreCase(row.getStatus())) {
result.put(row.getName(), Resolution.inactive());
} else {
- // STATUS_ACTIVE (default) or any non-INACTIVE status — treat as active substitution.
+ // STATUS_ACTIVE (default) or any non-INACTIVE status — active substitution.
final byte[] bytes = row.getContent() == null
? new byte[0]
: row.getContent().getBytes(StandardCharsets.UTF_8);
+ // Layer overrides are never permitted on top of a bundled rule. If the
+ // override carries layerDefinitions AND the disk twin exists, drop the
+ // override so the bundled content loads as-is. Operators get the boot
+ // error; the /addOrUpdate REST handler rejects the same combination
+ // pre-persist for fresh pushes.
+ if (isLayerOverrideOnBundled(catalog, row.getName(), bytes)) {
+ log.error("runtime-rule boot resolver: rule {}/{} declares layerDefinitions "
+ + "AND has a bundled disk twin. The substitution is dropped so the "
+ + "static loader serves the bundled disk content (bundled layer "
+ + "ownership preserved). The runtime row is still in the DAO and will "
+ + "be applied dynamically post-seal by RuleSync (its layerDefinitions "
+ + "go through the runtime channel and are operator-removable via "
+ + "/inactivate + /delete). Operator action: remove layerDefinitions "
+ + "from the runtime row, OR /delete the row and re-create it as a "
+ + "pure-runtime rule with a different name.",
+ catalog, row.getName());
+ continue;
+ }
result.put(row.getName(), Resolution.active(bytes));
}
}
log.info("Runtime-rule boot resolver loaded {} override(s) for catalog {}", result.size(), catalog);
return result;
}
+
+ /** True when content has a non-empty layerDefinitions block AND the disk twin
+ * exists. The static loader records every disk entry into {@link StaticRuleRegistry}
+ * before resolvers run, so the registry is authoritative for disk-twin presence. */
+ public static boolean isLayerOverrideOnBundled(final String catalog, final String name,
+ final byte[] bytes) {
+ if (bytes == null || bytes.length == 0) {
+ return false;
+ }
+ final StaticRuleRegistry registry = StaticRuleRegistry.active();
+ if (registry == null || !registry.find(catalog, name).isPresent()) {
+ return false;
+ }
+ return contentHasLayerDefinitions(catalog, new String(bytes, StandardCharsets.UTF_8));
+ }
+
+ /** YAML peek for the {@code layerDefinitions:} block. MAL catalogs bind into
+ * {@link Rule}; LAL into {@link LALConfigs}. Unknown catalogs return false (no
+ * catalog-specific parsing → can't enforce; caller treats as no layers). */
+ public static boolean contentHasLayerDefinitions(final String catalog, final String content) {
+ if (content == null || content.isEmpty()) {
+ return false;
+ }
+ try (StringReader reader = new StringReader(content)) {
+ final List<LayerDefinition> defs;
+ switch (catalog) {
+ case "otel-rules":
+ case "log-mal-rules":
+ case "telegraf-rules": {
+ final Rule rule = new Yaml().loadAs(reader, Rule.class);
+ defs = rule == null ? null : rule.getLayerDefinitions();
+ break;
+ }
+ case "lal": {
+ final LALConfigs configs = new Yaml().loadAs(reader, LALConfigs.class);
+ defs = configs == null ? null : configs.getLayerDefinitions();
+ break;
+ }
+ default:
+ return false;
+ }
+ return defs != null && !defs.isEmpty();
+ } catch (final RuntimeException ex) {
+ // YAML malformed: not our problem here. The applier surfaces the parse error
+ // with a clearer message; the layer-override check is a no-op for unparseable
+ // content (the apply will reject the row anyway).
+ return false;
+ }
+ }
}
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/AppliedClaims.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/AppliedClaims.java
new file mode 100644
index 0000000000..d6b2162553
--- /dev/null
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/AppliedClaims.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.receiver.runtimerule.layer;
+
+import java.util.Collections;
+import java.util.Set;
+import lombok.Getter;
+
+/**
+ * Snapshot of what a {@link RuntimeLayerRegistry#apply} call did, sufficient for
+ * {@link RuntimeLayerRegistry#rollback} to undo the change. Returned to the applier so it
+ * can hand the bundle to the orchestrator alongside the {@code Applied} artifact — the
+ * structural commit coordinator calls back into the registry with this token on persist /
+ * commit-tail failure.
+ *
+ * <p>{@link #getNewlyRegistered} is informational (which layers were brand-new to the
+ * registry, not just newly-claimed by this rule). Rollback uses the {@code priorClaims}
+ * and {@code currentLayerNames} diff to restore the exact state without needing the
+ * net-new set.
+ */
+@Getter
+public final class AppliedClaims {
+
+ private final String ruleId;
+ private final Set<LayerClaim> priorClaims;
+ private final Set<String> currentLayerNames;
+ private final Set<String> newlyRegistered;
+
+ AppliedClaims(final String ruleId,
+ final Set<LayerClaim> priorClaims,
+ final Set<String> currentLayerNames,
+ final Set<String> newlyRegistered) {
+ this.ruleId = ruleId;
+ this.priorClaims = priorClaims == null
+ ? Collections.emptySet()
+ : Collections.unmodifiableSet(priorClaims);
+ this.currentLayerNames = currentLayerNames == null
+ ? Collections.emptySet()
+ : Collections.unmodifiableSet(currentLayerNames);
+ this.newlyRegistered = newlyRegistered == null
+ ? Collections.emptySet()
+ : Collections.unmodifiableSet(newlyRegistered);
+ }
+
+ /** Convenience: true when the apply effected no actual change (empty old and new). */
+ public boolean isNoOp() {
+ return priorClaims.isEmpty() && currentLayerNames.isEmpty();
+ }
+}
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/LayerClaim.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/LayerClaim.java
new file mode 100644
index 0000000000..8b480d93d6
--- /dev/null
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/LayerClaim.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.receiver.runtimerule.layer;
+
+import java.util.Objects;
+import lombok.Getter;
+
+/**
+ * Immutable resolved layer declaration used inside the runtime-rule module. Produced from
+ * a MAL/LAL {@code layerDefinitions:} entry after the applier has resolved
+ * server-allocated ordinals (for entries with no explicit {@code ordinal:}) and validated
+ * the {@code >= RUNTIME_DYNAMIC_MIN_ORDINAL} floor.
+ *
+ * <p>Equality is by the full {@code (name, ordinal, normal)} triple — two claims are
+ * "same" iff all three fields match. This is the same notion of identity that
+ * {@link org.apache.skywalking.oap.server.core.analysis.Layer#registerDynamic} uses to
+ * decide whether re-registration is idempotent.
+ */
+@Getter
+public final class LayerClaim {
+
+ private final String name;
+ private final int ordinal;
+ private final boolean normal;
+
+ public LayerClaim(final String name, final int ordinal, final boolean normal) {
+ if (name == null) {
+ throw new IllegalArgumentException("layer name must not be null");
+ }
+ this.name = name;
+ this.ordinal = ordinal;
+ this.normal = normal;
+ }
+
+ @Override
+ public boolean equals(final Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof LayerClaim)) {
+ return false;
+ }
+ final LayerClaim that = (LayerClaim) o;
+ return ordinal == that.ordinal
+ && normal == that.normal
+ && name.equals(that.name);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(name, ordinal, normal);
+ }
+
+ @Override
+ public String toString() {
+ return "LayerClaim{name=" + name + ", ordinal=" + ordinal + ", normal=" + normal + "}";
+ }
+}
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/LayerConflictException.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/LayerConflictException.java
new file mode 100644
index 0000000000..017dcbf091
--- /dev/null
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/LayerConflictException.java
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.receiver.runtimerule.layer;
+
+import lombok.Getter;
+
+/**
+ * Typed exception raised by {@link RuntimeLayerConflictChecker} when a runtime DSL
+ * {@code layerDefinitions:} entry cannot be accepted. Carries a stable {@code applyStatus}
+ * code so the runtime-rule REST handler can map directly into the existing
+ * {@code {applyStatus, catalog, name, message}} response envelope without case analysis.
+ *
+ * <p>The {@code message} surfaces the offending value verbatim (layer name, conflicting
+ * ordinal, declaring source) so the operator's next action is obvious from the response
+ * body alone.
+ */
+@Getter
+public final class LayerConflictException extends RuntimeException {
+
+ /**
+ * Stable identifier mirrored as the {@code applyStatus} field on the HTTP response.
+ * Operators and automation should pattern-match on the enum constant name (which is
+ * exactly the wire value via {@link #wireValue()}).
+ */
+ public enum Status {
+ LAYER_ORDINAL_OUT_OF_RANGE("layer_ordinal_out_of_range"),
+ LAYER_NAME_CONFLICT("layer_name_conflict"),
+ LAYER_ORDINAL_COLLISION("layer_ordinal_collision"),
+ LAYER_NAME_INVALID("layer_name_invalid"),
+ /** Runtime override of a bundled rule attempted to declare layerDefinitions.
+ * Layer ownership lives with whichever channel registered it first; runtime
+ * overrides may change the rule body but must not touch layer declarations. */
+ LAYER_OVERRIDE_FORBIDDEN("layer_override_forbidden");
+
+ private final String wire;
+
+ Status(final String wire) {
+ this.wire = wire;
+ }
+
+ public String wireValue() {
+ return wire;
+ }
+ }
+
+ private final Status status;
+
+ public LayerConflictException(final Status status, final String message) {
+ super(message);
+ this.status = status;
+ }
+
+ /** Convenience accessor for callers building the HTTP response envelope. */
+ public String applyStatus() {
+ return status.wireValue();
+ }
+}
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/RuntimeLayerConflictChecker.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/RuntimeLayerConflictChecker.java
new file mode 100644
index 0000000000..65fa22a386
--- /dev/null
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/RuntimeLayerConflictChecker.java
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.receiver.runtimerule.layer;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import org.apache.skywalking.oap.server.core.analysis.Layer;
+
+/**
+ * Pure validation helper for runtime-DSL layer declarations. Produces operator-actionable
+ * {@link LayerConflictException}s that the REST handler translates 1:1 into the
+ * {@code {applyStatus, message}} response envelope. Every message names the offending
+ * layer, the conflicting ordinal, and (where applicable) the source that already owns the
+ * name, so the operator's corrective action is obvious from the response body alone.
+ *
+ * <p>Validation is performed against the live {@link Layer} registry, masking out the
+ * caller rule's prior claims so a same-rule edit (e.g. changing a runtime layer's ordinal
+ * from {@code 100_001} to {@code 100_002}) is permitted without an
+ * inactivate-then-recreate round trip.
+ */
+public final class RuntimeLayerConflictChecker {
+
+ /**
+ * Same regex enforced by {@link Layer#registerDynamic}; mirrored here so the operator
+ * gets the dedicated {@code layer_name_invalid} status (mapped to HTTP 400) instead
+ * of an {@code IllegalArgumentException} masquerading as a 500.
+ */
+ private static final Pattern NAME_PATTERN = Pattern.compile("[A-Z][A-Z0-9_]*");
+
+ private RuntimeLayerConflictChecker() {
+ }
+
+ /**
+ * Validate {@code candidates} against the current {@link Layer} registry and the
+ * given {@link RuntimeLayerRegistry}, treating the caller rule's {@code priorClaims}
+ * as already-removed (because the in-flight apply will replace them). Throws on the
+ * first detected conflict; partial applies are impossible because the caller invokes
+ * this BEFORE any {@link Layer#registerDynamic} call lands.
+ *
+ * @param ruleId canonical rule identifier — surfaced in conflict messages so
+ * the operator can pinpoint the source side of a cross-rule
+ * conflict
+ * @param candidates proposed layer declarations (ordinal already validated to
+ * {@code >= RUNTIME_DYNAMIC_MIN_ORDINAL} by the applier; this
+ * method re-checks for defence in depth)
+ * @param priorClaims layer names this rule currently claims; conflicts against
+ * these are NOT raised, since the apply will release them
+ * @param refcounts refcount tracker used to label runtime-side conflict sources
+ * ("runtime rule otel-rules/foo also declares X")
+ * @throws LayerConflictException on the first conflict
+ */
+ public static void validate(final String ruleId,
+ final List<LayerClaim> candidates,
+ final Set<String> priorClaims,
+ final RuntimeLayerRegistry refcounts) {
+ if (candidates == null || candidates.isEmpty()) {
+ return;
+ }
+ // Pre-pass: detect duplicates within the same batch BEFORE any registry check.
+ // Two entries naming the same layer with different triples, or two entries pinning
+ // the same ordinal under different names, cannot both succeed even in isolation;
+ // raising early keeps apply() atomic — without this pre-pass, a per-claim loop
+ // would mutate the registry for entry N before throwing on entry N+1, leaving a
+ // half-applied state that the rollback path then has to unwind.
+ final Map<String, LayerClaim> byName = new HashMap<>(candidates.size());
+ final Map<Integer, LayerClaim> byOrdinal = new HashMap<>(candidates.size());
+ for (final LayerClaim claim : candidates) {
+ final LayerClaim sameName = byName.get(claim.getName());
+ if (sameName != null && !sameName.equals(claim)) {
+ throw new LayerConflictException(
+ LayerConflictException.Status.LAYER_NAME_CONFLICT,
+ "Batch in rule " + ruleId + " declares layer '" + claim.getName()
+ + "' twice with different triples (" + sameName + " vs " + claim
+ + "). A single batch must declare each layer at most once.");
+ }
+ final LayerClaim sameOrdinal = byOrdinal.get(claim.getOrdinal());
+ if (sameOrdinal != null && !sameOrdinal.getName().equals(claim.getName())) {
+ throw new LayerConflictException(
+ LayerConflictException.Status.LAYER_ORDINAL_COLLISION,
+ "Batch in rule " + ruleId + " pins ordinal " + claim.getOrdinal()
+ + " under two different names ('" + sameOrdinal.getName()
+ + "' and '" + claim.getName() + "'). Each ordinal must be unique "
+ + "within a single layerDefinitions block.");
+ }
+ byName.put(claim.getName(), claim);
+ byOrdinal.put(claim.getOrdinal(), claim);
+ }
+ for (final LayerClaim claim : candidates) {
+ validateOne(ruleId, claim, priorClaims, refcounts);
+ }
+ }
+
+ private static void validateOne(final String ruleId,
+ final LayerClaim claim,
+ final Set<String> priorClaims,
+ final RuntimeLayerRegistry refcounts) {
+ // 1. Name shape.
+ if (claim.getName() == null || !NAME_PATTERN.matcher(claim.getName()).matches()) {
+ throw new LayerConflictException(
+ LayerConflictException.Status.LAYER_NAME_INVALID,
+ "Layer name '" + claim.getName() + "' is invalid — must match "
+ + "[A-Z][A-Z0-9_]* (upper-snake-case, leading letter, no spaces).");
+ }
+
+ // 2. Ordinal range.
+ if (claim.getOrdinal() < Layer.RUNTIME_DYNAMIC_MIN_ORDINAL) {
+ throw new LayerConflictException(
+ LayerConflictException.Status.LAYER_ORDINAL_OUT_OF_RANGE,
+ "Layer '" + claim.getName() + "' declared with ordinal " + claim.getOrdinal()
+ + " in rule " + ruleId + ". Runtime DSL layers must pin an explicit "
+ + "ordinal in the runtime tier (>= " + Layer.RUNTIME_DYNAMIC_MIN_ORDINAL
+ + "). Reserved ranges: 0-9999 = built-in Layer constants, "
+ + "10_000-99_999 = layer-extensions.yml + bundled MAL/LAL "
+ + "layerDefinitions. Omitting 'ordinal:' (default 0) is rejected "
+ + "because the ordinal is persisted in ServiceTraffic primary keys "
+ + "and must be operator-stable across restarts.");
+ }
+
+ // 3. Name conflict. Non-dynamic existing + same triple = soft claim (refcount
+ // tracks the runtime rule but Layer stays bundled). Anything else against a
+ // non-dynamic owner is a hard conflict. Against a dynamic owner, self-claims
+ // are permitted; cross-rule and multi-claimant triple changes are not.
+ final Layer byName = Layer.nameOf(claim.getName());
+ if (byName != Layer.UNDEFINED) {
+ final boolean tripleMatches =
+ byName.value() == claim.getOrdinal() && byName.isNormal() == claim.isNormal();
+ if (!Layer.isDynamic(claim.getName())) {
+ if (tripleMatches) {
+ // Soft claim — apply refcount-adds without calling registerDynamic.
+ return;
+ }
+ throw new LayerConflictException(
+ LayerConflictException.Status.LAYER_NAME_CONFLICT,
+ "Layer '" + claim.getName() + "' is already registered by "
+ + sourceLabel(claim.getName(), refcounts)
+ + " as (ordinal=" + byName.value()
+ + ", normal=" + byName.isNormal()
+ + ") through the boot-time channel; rule " + ruleId
+ + " declares (ordinal=" + claim.getOrdinal()
+ + ", normal=" + claim.isNormal() + "). Align the runtime "
+ + "declaration to match, or rename the runtime layer.");
+ }
+ if (!tripleMatches) {
+ final boolean priorlyClaimedByThisRule =
+ priorClaims != null && priorClaims.contains(claim.getName());
+ if (priorlyClaimedByThisRule) {
+ // Self-update is only legal if this rule is the sole claimant.
+ final Set<String> claimants = refcounts.claimantsOf(claim.getName());
+ if (claimants.size() > 1) {
+ throw new LayerConflictException(
+ LayerConflictException.Status.LAYER_NAME_CONFLICT,
+ "Layer '" + claim.getName() + "' is also claimed by other "
+ + "runtime rule(s) "
+ + String.join(", ", otherClaimants(claimants, ruleId))
+ + " at (ordinal=" + byName.value()
+ + ", normal=" + byName.isNormal()
+ + "); rule " + ruleId + " cannot change the triple to "
+ + "(ordinal=" + claim.getOrdinal()
+ + ", normal=" + claim.isNormal()
+ + ") because the change would affect the other claimants. "
+ + "Remove the prior declaration from this rule first, or "
+ + "have every claimant align on the new triple in lockstep.");
+ }
+ // Sole claimant — apply will replace the prior triple. Fall through.
+ } else {
+ throw new LayerConflictException(
+ LayerConflictException.Status.LAYER_NAME_CONFLICT,
+ "Layer '" + claim.getName() + "' already registered by "
+ + sourceLabel(claim.getName(), refcounts)
+ + " as (ordinal=" + byName.value()
+ + ", normal=" + byName.isNormal()
+ + "); rule " + ruleId + " declares (ordinal=" + claim.getOrdinal()
+ + ", normal=" + claim.isNormal()
+ + "). Align both declarations or rename one.");
+ }
+ } else {
+ // Same name and same triple — idempotent re-declare (dynamic only,
+ // because the !isDynamic branch above already rejected the bundled case).
+ return;
+ }
+ }
+
+ // 4. Ordinal collision — different name, same ordinal already used by someone.
+ // Sub-cases mirror step 3: bundled holders are immovable; dynamic holders allow
+ // a self-update where this rule is freeing the ordinal.
+ final Layer byOrdinal = Layer.peekByValue(claim.getOrdinal());
+ if (byOrdinal != null && !byOrdinal.name().equals(claim.getName())) {
+ if (!Layer.isDynamic(byOrdinal.name())) {
+ throw new LayerConflictException(
+ LayerConflictException.Status.LAYER_ORDINAL_COLLISION,
+ "Layer '" + claim.getName() + "' ordinal " + claim.getOrdinal()
+ + " is already used by layer '" + byOrdinal.name()
+ + "' (registered by " + sourceLabel(byOrdinal.name(), refcounts)
+ + " through the boot-time channel). Pick a different ordinal "
+ + "(>= " + Layer.RUNTIME_DYNAMIC_MIN_ORDINAL + ").");
+ }
+ final boolean priorlyHeldByThisRule =
+ priorClaims != null && priorClaims.contains(byOrdinal.name());
+ if (!priorlyHeldByThisRule) {
+ throw new LayerConflictException(
+ LayerConflictException.Status.LAYER_ORDINAL_COLLISION,
+ "Layer '" + claim.getName() + "' ordinal " + claim.getOrdinal()
+ + " is already used by layer '" + byOrdinal.name()
+ + "' (registered by " + sourceLabel(byOrdinal.name(), refcounts)
+ + "). Pick a different ordinal (>= "
+ + Layer.RUNTIME_DYNAMIC_MIN_ORDINAL + ").");
+ }
+ // Self-update freeing the ordinal is only legal if this rule is the sole
+ // claimant of the prior layer. Other claimants would still hold the slot
+ // after apply Pass-1 drops THIS rule's claim, so registerDynamic would
+ // throw an unstructured collision deep inside the apply path.
+ final Set<String> otherHolders = refcounts.claimantsOf(byOrdinal.name());
+ if (otherHolders.size() > 1) {
+ throw new LayerConflictException(
+ LayerConflictException.Status.LAYER_ORDINAL_COLLISION,
+ "Layer '" + claim.getName() + "' ordinal " + claim.getOrdinal()
+ + " is still held by layer '" + byOrdinal.name()
+ + "' under other runtime rule(s) "
+ + String.join(", ", otherClaimants(otherHolders, ruleId))
+ + "; rule " + ruleId + " cannot reclaim the ordinal alone. "
+ + "Either drop the prior layer from this rule with the others "
+ + "in lockstep, or pick a different ordinal.");
+ }
+ }
+ }
+
+ /** Filter {@code claimants} to entries other than {@code self}. Used in self-update
+ * conflict messages to name only the rules whose declaration would be impacted. */
+ private static List<String> otherClaimants(final Set<String> claimants, final String self) {
+ final List<String> out = new ArrayList<>(claimants.size());
+ for (final String c : claimants) {
+ if (!c.equals(self)) {
+ out.add(c);
+ }
+ }
+ return out;
+ }
+
+ /**
+ * Label the source that owns a layer's current registration. Granularity follows the
+ * {@link Layer#isDynamic} channel, NOT the ordinal range — operator yaml may put a
+ * boot-time layer anywhere the registry accepts, including the {@code >=100_000}
+ * tier; labelling those as "runtime DSL" would mislead the operator into thinking
+ * they could realign at runtime.
+ * <ul>
+ * <li>{@code "built-in or boot-time extension"} — registered through
+ * {@link Layer#register} (built-in {@code Layer.*} constants,
+ * {@code layer-extensions.yml}, bundled MAL/LAL {@code layerDefinitions:}).</li>
+ * <li>{@code "runtime rule(s) <id1>, <id2>, ..."} — registered through
+ * {@link Layer#registerDynamic}, listing every runtime rule currently claiming
+ * the name through the refcount tracker.</li>
+ * </ul>
+ */
+ private static String sourceLabel(final String name, final RuntimeLayerRegistry refcounts) {
+ final Layer layer = Layer.nameOf(name);
+ if (layer == Layer.UNDEFINED) {
+ return "(unknown source — registry inconsistency)";
+ }
+ if (!Layer.isDynamic(name)) {
+ return "built-in or boot-time extension";
+ }
+ final Set<String> claimants = refcounts.claimantsOf(name);
+ if (claimants.isEmpty()) {
+ // Edge case: layer is in the runtime tier but not tracked by refcount. Should
+ // not happen under the synchronized boundary; log-and-fall-back if it does.
+ return "runtime DSL (untracked)";
+ }
+ return "runtime rule(s) " + String.join(", ", claimants);
+ }
+}
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/RuntimeLayerRegistry.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/RuntimeLayerRegistry.java
new file mode 100644
index 0000000000..ddb0d310f7
--- /dev/null
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/RuntimeLayerRegistry.java
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.receiver.runtimerule.layer;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.analysis.Layer;
+
+/**
+ * In-memory refcount tracker for runtime-DSL dynamic layer declarations. Sits between the
+ * runtime-rule appliers and {@link Layer}'s post-seal {@link Layer#registerDynamic} /
+ * {@link Layer#unregisterDynamic} entry points.
+ *
+ * <p>Lifecycle invariant: a layer is present in {@link Layer}'s registry (with ordinal
+ * {@code >= RUNTIME_DYNAMIC_MIN_ORDINAL}) iff at least one runtime rule currently claims
+ * it through this registry. When the last claim drops, the layer is unregistered. Layers
+ * with ordinals below {@link Layer#RUNTIME_DYNAMIC_MIN_ORDINAL} are never tracked here —
+ * they are owned by the bundled / boot-time channels and are non-removable.
+ *
+ * <p>Crash recovery: state lives in-memory only. On OAP restart the runtime-rule replay
+ * path re-applies every stored runtime rule through this registry, rebuilding the same
+ * refcount snapshot. No separate persistence is needed because the rule yaml itself is
+ * the source of truth for layer declarations.
+ *
+ * <p>Cluster semantics: each OAP instance owns an independent {@code RuntimeLayerRegistry}.
+ * Convergence comes from operator-pinned ordinals (same yaml on every node produces the
+ * same registration) plus rule yaml replicating identically through the existing
+ * runtime-rule storage. Refcount state is process-local — peers don't share the map; they
+ * each track their own claims.
+ *
+ * <p>Thread safety: every public method is {@code synchronized} so apply / rollback /
+ * teardown sequences are atomic. The registry is the single mutator of {@link Layer}'s
+ * dynamic entries inside the runtime-rule module — the appliers and orchestrator must not
+ * call {@link Layer#registerDynamic} directly.
+ */
+@Slf4j
+public final class RuntimeLayerRegistry {
+
+ /**
+ * Process-wide singleton mirroring {@link Layer}'s process-singleton scope. Used by the
+ * MAL/LAL engines from their compile / rollback / unregister paths. Tests that need
+ * an isolated registry can construct one via {@link #RuntimeLayerRegistry()}.
+ */
+ public static final RuntimeLayerRegistry INSTANCE = new RuntimeLayerRegistry();
+
+ /**
+ * Claim graph: layer name → set of rule IDs that have declared this layer. A layer's
+ * lifetime in {@link Layer}'s registry is bounded by this set being non-empty. Use
+ * {@link LinkedHashSet} so iteration order is stable for diagnostic output.
+ */
+ private final Map<String, Set<String>> claimsByLayer = new HashMap<>();
+
+ /**
+ * Reverse index: rule ID → set of layer names this rule currently claims. Maintained
+ * in lockstep with {@link #claimsByLayer} so a rule's prior claims can be looked up
+ * in {@code O(claims)} during UPDATE.
+ */
+ private final Map<String, Set<String>> claimsByRule = new HashMap<>();
+
+ /**
+ * Static helper for building the canonical rule ID consumed by every method on this
+ * registry. The MAL/LAL appliers receive {@code sourceName} as {@code "catalog/name"}
+ * already, so the simplest path is to pass that string through verbatim — keeping the
+ * registry decoupled from the {@code Catalog} enum.
+ */
+ public static String ruleId(final String catalog, final String name) {
+ return catalog + "/" + name;
+ }
+
+ /**
+ * Validate the proposed declarations against the current state, then apply them
+ * atomically: register net-new layers through {@link Layer#registerDynamic}, replace
+ * this rule's prior claims with {@code newClaims} in the refcount graph, and
+ * unregister any layer whose last claimant just dropped.
+ *
+ * <p>Atomicity is per-rule: if validation throws partway, no state changes are
+ * committed. If a {@link Layer#registerDynamic} call throws after some registrations
+ * have already landed (race against another OAP-internal mutator — should not happen
+ * under the synchronized boundary, defensive only), the partial state is rolled back
+ * before re-throwing.
+ *
+ * @param ruleId canonical rule identifier (use {@link #ruleId(String, String)})
+ * @param newClaims the resolved layer declarations from the rule yaml (post-allocation,
+ * post-validation of the {@code >= 100_000} ordinal floor)
+ * @return {@link AppliedClaims} carrying the prior state so the caller can roll back
+ * later (e.g. on persist / commit failure outside the applier).
+ * @throws LayerConflictException on any name/ordinal conflict against the current
+ * registry view (excluding this rule's own prior claims, so self-edits work).
+ */
+ public synchronized AppliedClaims apply(final String ruleId, final List<LayerClaim> newClaims) {
+ if (ruleId == null || ruleId.isEmpty()) {
+ throw new IllegalArgumentException("ruleId must be non-empty");
+ }
+ final Set<String> priorLayerNames = new LinkedHashSet<>(
+ claimsByRule.getOrDefault(ruleId, Collections.emptySet()));
+ // Snapshot BEFORE mutation — pass-1 removeClaim may unregister entries that
+ // priorLayerNamesSnapshot would no longer be able to resolve.
+ final Set<LayerClaim> priorClaimsSnapshot = priorLayerNamesSnapshot(priorLayerNames);
+
+ if (newClaims != null && !newClaims.isEmpty()) {
+ RuntimeLayerConflictChecker.validate(ruleId, newClaims, priorLayerNames, this);
+ }
+
+ // Diff plan computed up front so the two passes below are straight-line.
+ final Set<String> newLayerNames = new LinkedHashSet<>();
+ final Map<String, LayerClaim> newByName = new LinkedHashMap<>();
+ if (newClaims != null) {
+ for (final LayerClaim claim : newClaims) {
+ newLayerNames.add(claim.getName());
+ newByName.put(claim.getName(), claim);
+ }
+ }
+ final Set<String> toRemoveNames = new LinkedHashSet<>();
+ for (final LayerClaim prior : priorClaimsSnapshot) {
+ final LayerClaim incoming = newByName.get(prior.getName());
+ if (incoming == null || !incoming.equals(prior)) {
+ toRemoveNames.add(prior.getName());
+ }
+ }
+ final List<LayerClaim> toRegister = new ArrayList<>();
+ for (final LayerClaim claim : newByName.values()) {
+ final Layer existing = Layer.nameOf(claim.getName());
+ final boolean sameLiveTriple = existing != Layer.UNDEFINED
+ && existing.value() == claim.getOrdinal()
+ && existing.isNormal() == claim.isNormal();
+ if (sameLiveTriple) {
+ // Soft-claim case (existing is bundled) or dynamic-and-same-triple — no
+ // Layer mutation needed; refcount-add still happens in pass 2.
+ continue;
+ }
+ toRegister.add(claim);
+ }
+
+ // Pass 1: drop prior claims. Frees ordinals so swap-style replacements in pass 2
+ // can register against the now-free slot.
+ for (final String name : toRemoveNames) {
+ removeClaim(ruleId, name);
+ }
+
+ final Set<String> newlyRegisteredInThisApply = new LinkedHashSet<>();
+ try {
+ for (final LayerClaim claim : toRegister) {
+ final boolean wasAbsent = Layer.nameOf(claim.getName()) == Layer.UNDEFINED;
+ Layer.registerDynamic(claim.getName(), claim.getOrdinal(), claim.isNormal());
+ if (wasAbsent) {
+ newlyRegisteredInThisApply.add(claim.getName());
+ }
+ }
+ for (final LayerClaim claim : newClaims == null ? Collections.<LayerClaim>emptyList() : newClaims) {
+ claimsByLayer.computeIfAbsent(claim.getName(), k -> new LinkedHashSet<>())
+ .add(ruleId);
+ }
+ } catch (final RuntimeException e) {
+ // Defence-in-depth: synchronized + post-validate should make this unreachable.
+ for (final String name : newlyRegisteredInThisApply) {
+ try {
+ Layer.unregisterDynamic(name);
+ } catch (final RuntimeException ignore) {
+ }
+ }
+ for (final LayerClaim prior : priorClaimsSnapshot) {
+ try {
+ Layer.registerDynamic(prior.getName(), prior.getOrdinal(), prior.isNormal());
+ } catch (final RuntimeException ignore) {
+ }
+ claimsByLayer.computeIfAbsent(prior.getName(), k -> new LinkedHashSet<>())
+ .add(ruleId);
+ }
+ // Restore reverse index.
+ claimsByRule.put(ruleId, new LinkedHashSet<>(priorLayerNames));
+ throw e;
+ }
+
+ // Reverse index — final state matches new claims; intermediate Pass-1 mutations
+ // already cleaned out the prior names.
+ if (newLayerNames.isEmpty()) {
+ claimsByRule.remove(ruleId);
+ } else {
+ claimsByRule.put(ruleId, newLayerNames);
+ }
+
+ return new AppliedClaims(ruleId, priorClaimsSnapshot, newLayerNames,
+ newlyRegisteredInThisApply);
+ }
+
+ /**
+ * Undo the effect of an earlier {@link #apply(String, List)}: restore this rule's
+ * prior claims and unregister any layer that had been registered net-new by the apply
+ * and is no longer claimed.
+ *
+ * <p>Used by the orchestrator on persist / commit failure (after the applier had
+ * already succeeded). For in-applier failures the apply method's own internal
+ * rollback handles cleanup before re-throwing — callers do NOT need to invoke this
+ * for that case.
+ *
+ * @param applied bundle returned from the matching {@link #apply} call
+ */
+ public synchronized void rollback(final AppliedClaims applied) {
+ if (applied == null) {
+ return;
+ }
+ final String ruleId = applied.getRuleId();
+
+ // Drop the newly-claimed entries that the apply added.
+ for (final String name : applied.getCurrentLayerNames()) {
+ removeClaim(ruleId, name);
+ }
+
+ // Restore prior claims (re-registering layers if the apply had freed them).
+ for (final LayerClaim priorClaim : applied.getPriorClaims()) {
+ try {
+ Layer.registerDynamic(priorClaim.getName(), priorClaim.getOrdinal(),
+ priorClaim.isNormal());
+ } catch (final RuntimeException e) {
+ // Best-effort: the prior claim's layer might already be registered
+ // (idempotent re-register handles same-triple). Log and continue so the
+ // refcount graph stays correct even if one entry fails to re-register.
+ log.warn("runtime-layer rollback: failed to re-register prior claim {}",
+ priorClaim, e);
+ }
+ claimsByLayer.computeIfAbsent(priorClaim.getName(), k -> new LinkedHashSet<>())
+ .add(ruleId);
+ claimsByRule.computeIfAbsent(ruleId, k -> new LinkedHashSet<>())
+ .add(priorClaim.getName());
+ }
+
+ if (!claimsByRule.containsKey(ruleId)
+ || claimsByRule.get(ruleId).isEmpty()) {
+ claimsByRule.remove(ruleId);
+ }
+ }
+
+ /**
+ * Drop every claim owned by {@code ruleId}. Called by the unregister path on DELETE /
+ * INACTIVATE — the rule is gone, so its layers must release their refcount slot, and
+ * any layer whose last claimant just dropped is unregistered through
+ * {@link Layer#unregisterDynamic}.
+ */
+ public synchronized void removeRule(final String ruleId) {
+ final Set<String> owned = claimsByRule.remove(ruleId);
+ if (owned == null || owned.isEmpty()) {
+ return;
+ }
+ for (final String name : owned) {
+ final Set<String> claimants = claimsByLayer.get(name);
+ if (claimants == null) {
+ continue;
+ }
+ claimants.remove(ruleId);
+ if (claimants.isEmpty()) {
+ claimsByLayer.remove(name);
+ if (Layer.isDynamic(name)) {
+ try {
+ Layer.unregisterDynamic(name);
+ } catch (final RuntimeException e) {
+ log.warn("runtime-layer removeRule: unregister threw for {} "
+ + "(unexpected — Layer.isDynamic was true)", name, e);
+ }
+ }
+ // else: soft claim (bundled-tier owner). Refcount entry dropped; the
+ // bundled registration stays per the layer's lifecycle contract.
+ }
+ }
+ }
+
+ /**
+ * Snapshot of layer names this rule currently claims. Returns an empty unmodifiable
+ * set if the rule has no claims. Used by the conflict checker to exclude self-edits
+ * from conflict probes.
+ */
+ public synchronized Set<String> claimsOf(final String ruleId) {
+ final Set<String> owned = claimsByRule.get(ruleId);
+ if (owned == null || owned.isEmpty()) {
+ return Collections.emptySet();
+ }
+ return Collections.unmodifiableSet(new LinkedHashSet<>(owned));
+ }
+
+ /**
+ * Snapshot of all currently-claimed runtime dynamic layers. The map key is the layer
+ * name; the value is the set of rules that have declared it. Used by the conflict
+ * checker for source labelling ("runtime rule otel-rules/foo also declares this").
+ */
+ public synchronized Map<String, Set<String>> snapshot() {
+ final Map<String, Set<String>> copy = new LinkedHashMap<>();
+ for (final Map.Entry<String, Set<String>> e : claimsByLayer.entrySet()) {
+ copy.put(e.getKey(), Collections.unmodifiableSet(new LinkedHashSet<>(e.getValue())));
+ }
+ return Collections.unmodifiableMap(copy);
+ }
+
+ /** Lookup helper for the conflict checker: which rules claim this layer right now? */
+ synchronized Set<String> claimantsOf(final String layerName) {
+ final Set<String> claimants = claimsByLayer.get(layerName);
+ if (claimants == null) {
+ return Collections.emptySet();
+ }
+ return Collections.unmodifiableSet(new LinkedHashSet<>(claimants));
+ }
+
+ /** Remove {@code ruleId} from {@code layerName}'s claimants and from the reverse index;
+ * if the claimants set empties, delete the entry and unregister the layer IF the
+ * layer was registered through the dynamic channel. Soft claims (a runtime rule
+ * whose triple matches an existing bundled / boot-time layer — see the conflict
+ * checker's case 3a) do NOT unregister: the bundled tier owns the layer's lifecycle
+ * and only a restart-time removal of the boot-time source can drop it. */
+ private void removeClaim(final String ruleId, final String layerName) {
+ final Set<String> claimants = claimsByLayer.get(layerName);
+ if (claimants != null) {
+ claimants.remove(ruleId);
+ if (claimants.isEmpty()) {
+ claimsByLayer.remove(layerName);
+ if (Layer.isDynamic(layerName)) {
+ try {
+ Layer.unregisterDynamic(layerName);
+ } catch (final RuntimeException e) {
+ log.warn("runtime-layer apply: unregister threw for {} (unexpected — "
+ + "Layer.isDynamic was true)", layerName, e);
+ }
+ }
+ // else: non-dynamic layer (bundled / boot-time extension). Soft claim
+ // released; layer stays in the registry per the bundled-tier contract.
+ }
+ }
+ // Keep the reverse index in lockstep — without this, rollback re-adds prior
+ // claims on top of the (still-stale) currentLayerNames entries.
+ final Set<String> owned = claimsByRule.get(ruleId);
+ if (owned != null) {
+ owned.remove(layerName);
+ if (owned.isEmpty()) {
+ claimsByRule.remove(ruleId);
+ }
+ }
+ }
+
+ /**
+ * Re-resolve full prior claims (with ordinal/normal) from layer names. Used in
+ * {@link #apply} to build the prior-state component of {@link AppliedClaims} so
+ * {@link #rollback} can restore the exact triples. Walks {@link Layer}'s registry
+ * (lookups by name) because the registry is the live source of truth for
+ * {@code (ordinal, normal)}.
+ */
+ private Set<LayerClaim> priorLayerNamesSnapshot(final Set<String> priorLayerNames) {
+ if (priorLayerNames.isEmpty()) {
+ return Collections.emptySet();
+ }
+ final Set<LayerClaim> out = new LinkedHashSet<>();
+ for (final String name : priorLayerNames) {
+ final Layer existing = Layer.nameOf(name);
+ if (existing == Layer.UNDEFINED) {
+ // Shouldn't happen — we tracked this rule as claiming `name`, so it must
+ // have been registered. Skip rather than throw so rollback can proceed
+ // for the other entries.
+ log.warn("runtime-layer snapshot: prior-claimed layer {} not in registry "
+ + "(skipping during rollback bundle)", name);
+ continue;
+ }
+ out.add(new LayerClaim(existing.name(), existing.value(), existing.isNormal()));
+ }
+ return Collections.unmodifiableSet(out);
+ }
+}
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/reconcile/DSLRuntimeApply.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/reconcile/DSLRuntimeApply.java
index 31a0178287..3d924d37f6 100644
--- a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/reconcile/DSLRuntimeApply.java
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/reconcile/DSLRuntimeApply.java
@@ -165,6 +165,12 @@ public final class DSLRuntimeApply {
final CompiledDSL compiled;
try {
compiled = engine.compile(file, classification, kind, ctx);
+ } catch (final org.apache.skywalking.oap.server.receiver.runtimerule.layer.LayerConflictException lce) {
+ // Layer-declaration conflicts (operator-actionable, structured 400 at the REST
+ // layer) bypass the generic compile-failed path so RuntimeRuleService can
+ // surface the typed applyStatus + message. Re-thrown unchanged; the REST
+ // handler's catch already maps to badRequest with the structured envelope.
+ throw lce;
} catch (final EngineCompileException ece) {
log.error("runtime-rule apply: compile FAILED for {}/{}: {}",
file.getCatalog(), file.getName(), ece.getMessage(), ece);
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/rest/RuntimeRuleService.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/rest/RuntimeRuleService.java
index ff2f99475a..01d645a4e8 100644
--- a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/rest/RuntimeRuleService.java
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/rest/RuntimeRuleService.java
@@ -65,6 +65,8 @@ import org.apache.skywalking.oap.server.receiver.runtimerule.apply.DSLDelta;
import org.apache.skywalking.oap.server.receiver.runtimerule.apply.DeltaClassifier;
import org.apache.skywalking.oap.server.receiver.runtimerule.engine.Classification;
import org.apache.skywalking.oap.server.receiver.runtimerule.engine.RuleEngine;
+import org.apache.skywalking.oap.server.receiver.runtimerule.extension.DbOverrideRuntimeRuleResolver;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.LayerConflictException;
import org.apache.skywalking.oap.server.receiver.runtimerule.cluster.MainRouter;
import org.apache.skywalking.oap.server.receiver.runtimerule.cluster.RuntimeRuleClusterClient;
import org.apache.skywalking.oap.server.receiver.runtimerule.cluster.v1.ForwardResponse;
@@ -719,6 +721,19 @@ public class RuntimeRuleService {
if (content.isEmpty()) {
return badRequest("empty_body", catalog, name, "request body must be the raw rule content");
}
+ // Runtime overrides of bundled rules MUST NOT declare layerDefinitions. Layer
+ // ownership is single-source-of-truth — bundled rules own bundled-tier layers,
+ // runtime rules own runtime-tier layers, no overwrite either way. Reject pre-
+ // persist so the bad row never reaches the DAO.
+ if (DbOverrideRuntimeRuleResolver.isLayerOverrideOnBundled(
+ catalog, name, content.getBytes(StandardCharsets.UTF_8))) {
+ return badRequest("layer_override_forbidden", catalog, name,
+ "Runtime override of bundled rule " + catalog + "/" + name + " declared "
+ + "layerDefinitions. Bundled rules own their layer declarations; "
+ + "runtime overrides may only change the rule body. Remove the "
+ + "layerDefinitions block, or rename the rule so it becomes a pure-"
+ + "runtime rule (no bundled twin).");
+ }
// Single-main routing. Self is main → null → run local workflow. Non-main + not
// forwarded → forward to main and relay response. Non-main + forwarded → fail-safe
// 421 (cluster view split; refuse to re-forward).
@@ -900,6 +915,19 @@ public class RuntimeRuleService {
try {
postApply = dslManager.applyNowForRuleFile(ruleFile);
} catch (final Throwable t) {
+ // Layer conflict on FILTER_ONLY is rare — DeltaClassifier escalates
+ // layerDefinitions changes to STRUCTURAL precisely so this path doesn't see
+ // them post-persist. The defensive unwrap-and-surface guards against a
+ // classifier miss (e.g. yaml-key reorder that doesn't change the signature
+ // but bypasses the equality check); operators still see the structured 400.
+ final LayerConflictException lce = findLayerConflict(t);
+ if (lce != null) {
+ log.error("runtime-rule FILTER_ONLY apply hit a layer conflict POST-persist "
+ + "for {}/{} — the row is in DB but local apply was rejected. Operator "
+ + "must reconcile (DELETE + re-POST with aligned layerDefinitions). "
+ + "Conflict: {}", catalog, name, lce.getMessage());
+ return badRequest(lce.applyStatus(), catalog, name, lce.getMessage());
+ }
log.error("runtime-rule FILTER_ONLY apply failed after persist for {}/{} — DB "
+ "reflects the new content; this node will converge on the next dslManager "
+ "tick (same path peers use).", catalog, name, t);
@@ -980,7 +1008,27 @@ public class RuntimeRuleService {
final DSLRuntimeState postApply;
try {
postApply = dslManager.applyNowForRuleFile(ruleFile, true);
+ } catch (final LayerConflictException lce) {
+ // Runtime-DSL layer-declaration conflict — operator-actionable, not a server
+ // error. Resume local + peers immediately so the cluster keeps serving the
+ // prior bundle.
+ log.warn("runtime-rule STRUCTURAL apply rejected for {}/{} — layer conflict: {}",
+ catalog, name, lce.getMessage());
+ dslManager.getSuspendCoord().localResume(catalog, name);
+ broadcastResume(catalog, name, "layer_conflict");
+ return badRequest(lce.applyStatus(), catalog, name, lce.getMessage());
} catch (final Throwable t) {
+ // Some exceptions wrap LayerConflictException as cause; unwrap so the operator
+ // still sees the structured 400 response. We probe to a small depth to avoid
+ // recursing through pathological cause chains.
+ final LayerConflictException wrapped = findLayerConflict(t);
+ if (wrapped != null) {
+ log.warn("runtime-rule STRUCTURAL apply rejected for {}/{} — layer conflict "
+ + "(wrapped): {}", catalog, name, wrapped.getMessage());
+ dslManager.getSuspendCoord().localResume(catalog, name);
+ broadcastResume(catalog, name, "layer_conflict");
+ return badRequest(wrapped.applyStatus(), catalog, name, wrapped.getMessage());
+ }
log.error("runtime-rule STRUCTURAL apply threw for {}/{}", catalog, name, t);
dslManager.getSuspendCoord().localResume(catalog, name);
// Peers went SUSPENDED on our earlier broadcast; let them know the apply
@@ -1741,6 +1789,24 @@ public class RuntimeRuleService {
jsonBody(applyStatus, catalog, name, message));
}
+ /**
+ * Walk up to 8 levels of cause chain looking for a {@link LayerConflictException}. The
+ * DSL engines wrap exceptions in {@code EngineCompileException} / {@code ApplyException},
+ * which in turn may be wrapped by infrastructure exceptions; surfacing the conflict's
+ * structured status / message regardless of wrapping depth keeps the operator-facing
+ * envelope consistent with the design intent that conflicts ARE bad-request shaped.
+ */
+ private static LayerConflictException findLayerConflict(final Throwable top) {
+ Throwable cursor = top;
+ for (int i = 0; cursor != null && i < 8; i++) {
+ if (cursor instanceof LayerConflictException) {
+ return (LayerConflictException) cursor;
+ }
+ cursor = cursor.getCause();
+ }
+ return null;
+ }
+
private static HttpResponse serverError(final String applyStatus, final String catalog,
final String name, final String message) {
return HttpResponse.of(HttpStatus.INTERNAL_SERVER_ERROR, MediaType.JSON_UTF_8,
diff --git a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/state/EngineApplied.java b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/state/EngineApplied.java
index c2fe3f49fa..8e9c269380 100644
--- a/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/state/EngineApplied.java
+++ b/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/state/EngineApplied.java
@@ -20,6 +20,7 @@ package org.apache.skywalking.oap.server.receiver.runtimerule.state;
import java.util.Set;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.AppliedClaims;
/**
* Engine-opaque per-rule applied artefact slot on {@link AppliedRuleScript}. Every engine's
@@ -93,4 +94,20 @@ public interface EngineApplied {
* @return immutable, possibly empty set of metric names
*/
Set<String> alarmResetTargets();
+
+ /**
+ * Snapshot of the {@link AppliedClaims} produced by this apply's interaction with the
+ * runtime-layer registry, or {@code null} when the apply declared no
+ * {@code layerDefinitions:} (the common case for bundled / non-layer rules). The
+ * structural commit coordinator calls
+ * {@link org.apache.skywalking.oap.server.receiver.runtimerule.layer.RuntimeLayerRegistry#rollback}
+ * with this token on a persist / commit-tail failure that lands outside the applier's
+ * own try/catch.
+ *
+ * <p>Default returns {@code null} so existing engines (and any future engines that
+ * do not own layer state) need no implementation change.
+ */
+ default AppliedClaims appliedLayerClaims() {
+ return null;
+ }
}
diff --git a/oap-server/server-admin/runtime-rule/src/test/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/MalFileApplierTest.java b/oap-server/server-admin/runtime-rule/src/test/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/MalFileApplierTest.java
index 726b1751ae..dc4222b491 100644
--- a/oap-server/server-admin/runtime-rule/src/test/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/MalFileApplierTest.java
+++ b/oap-server/server-admin/runtime-rule/src/test/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/MalFileApplierTest.java
@@ -22,8 +22,11 @@ import java.util.Collections;
import java.util.HashSet;
import java.util.Set;
import javassist.ClassPool;
+import org.apache.skywalking.oap.server.core.analysis.Layer;
import org.apache.skywalking.oap.server.core.analysis.meter.MeterSystem;
import org.apache.skywalking.oap.server.core.storage.model.StorageManipulationOpt;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.LayerConflictException;
+import org.apache.skywalking.oap.server.receiver.runtimerule.layer.RuntimeLayerRegistry;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.mockito.Mockito;
@@ -207,6 +210,84 @@ class MalFileApplierTest {
assertEquals(setOf("meter_x_one"), applied.getRegisteredMetricNames());
}
+ @Test
+ void applyWithLayerDefinitionsRegistersLayerAndCarriesAppliedClaims() throws Exception {
+ // Happy path: a runtime MAL rule that declares a new layer in layerDefinitions
+ // with a properly pinned ordinal in the runtime tier (>=100_000) must:
+ // (a) register the layer through the dynamic channel (Layer.isDynamic = true);
+ // (b) carry the AppliedClaims token on the returned Applied artifact so the
+ // structural commit coordinator can roll it back on persist failure;
+ // (c) populate the layer-registry refcount with this rule as the claimant.
+ // Uses an isolated RuntimeLayerRegistry to avoid touching JVM-wide state.
+ // `normal:` is intentionally omitted to exercise the LayerDefinition default
+ // (true) — same operator ergonomics the bundled yaml + layer-extensions tier
+ // already enjoy.
+ final RuntimeLayerRegistry isolatedRegistry = new RuntimeLayerRegistry();
+ final MalFileApplier layerApplier = new MalFileApplier(meterSystem, isolatedRegistry);
+ final String yaml =
+ "metricPrefix: meter_hl\n"
+ + "layerDefinitions:\n"
+ + " - name: MAL_APPLIER_HAPPY_LAYER\n"
+ + " ordinal: 100600\n"
+ + "expSuffix: service(['host'], Layer.OS_LINUX)\n"
+ + "metricsRules:\n"
+ + " - name: ok\n"
+ + " exp: m.sum(['host'])\n";
+ final MalFileApplier.Applied applied =
+ layerApplier.apply(yaml, "otel-rules/happy_layer_mal", "hashHL");
+ try {
+ // (a) layer registered through the dynamic channel with the default normal
+ assertEquals(100_600, Layer.nameOf("MAL_APPLIER_HAPPY_LAYER").value());
+ assertTrue(Layer.nameOf("MAL_APPLIER_HAPPY_LAYER").isNormal(),
+ "omitted normal: defaults to true, same as the other tiers");
+ assertTrue(Layer.isDynamic("MAL_APPLIER_HAPPY_LAYER"));
+ // (b) AppliedClaims carried for orchestrator rollback
+ assertNotNull(applied.appliedLayerClaims());
+ assertTrue(applied.appliedLayerClaims().getCurrentLayerNames()
+ .contains("MAL_APPLIER_HAPPY_LAYER"));
+ assertTrue(applied.appliedLayerClaims().getNewlyRegistered()
+ .contains("MAL_APPLIER_HAPPY_LAYER"));
+ // (c) refcount tracks this rule as the claimant
+ assertTrue(isolatedRegistry.snapshot().get("MAL_APPLIER_HAPPY_LAYER")
+ .contains("otel-rules/happy_layer_mal"));
+ } finally {
+ // Clean up so JVM-wide Layer state stays sane for sibling tests.
+ isolatedRegistry.removeRule("otel-rules/happy_layer_mal");
+ }
+ }
+
+ @Test
+ void applyWithLayerOrdinalBelowFloorThrowsLayerConflict() {
+ // Sad path mirror of the happy-path test: ordinal=99_999 (the last ordinal of
+ // the boot-time external tier — one slot below the 100_000 runtime floor) must
+ // be rejected with layer_ordinal_out_of_range. The boundary value is the most
+ // realistic operator mistake: the operator remembers "external layers go in
+ // 10_000-99_999" and reuses that convention for a runtime rule. The applier
+ // wraps the LayerConflictException through; the REST handler maps it to HTTP
+ // 400 with the structured envelope. This proves the validation gate fires
+ // BEFORE compile so no MeterSystem.create runs for a rule we're going to
+ // reject.
+ final RuntimeLayerRegistry isolatedRegistry = new RuntimeLayerRegistry();
+ final MalFileApplier layerApplier = new MalFileApplier(meterSystem, isolatedRegistry);
+ final String yaml =
+ "metricPrefix: meter_bad_floor\n"
+ + "layerDefinitions:\n"
+ + " - name: MAL_APPLIER_BAD_FLOOR\n"
+ + " ordinal: 99999\n"
+ + "expSuffix: service(['host'], Layer.OS_LINUX)\n"
+ + "metricsRules:\n"
+ + " - name: ok\n"
+ + " exp: m.sum(['host'])\n";
+ final LayerConflictException ex = assertThrows(
+ LayerConflictException.class,
+ () -> layerApplier.apply(yaml, "otel-rules/bad_floor", "h"));
+ assertEquals(LayerConflictException.Status.LAYER_ORDINAL_OUT_OF_RANGE, ex.getStatus());
+ // MeterSystem.create was never invoked because validation fired first.
+ verify(meterSystem, Mockito.never())
+ .create(anyString(), anyString(), any(), any(ClassPool.class), any(ClassLoader.class),
+ any(StorageManipulationOpt.class));
+ }
+
@Test
void legacyTwoArgApplyUsesEmptyHash() throws Exception {
// Back-compat overload — loader identity still constructible, just with a less
diff --git a/oap-server/server-admin/runtime-rule/src/test/java/org/apache/skywalking/oap/server/receiver/runtimerule/extension/DbOverrideRuntimeRuleResolverTest.java b/oap-server/server-admin/runtime-rule/src/test/java/org/apache/skywalking/oap/server/receiver/runtimerule/extension/DbOverrideRuntimeRuleResolverTest.java
new file mode 100644
index 0000000000..59a07958ad
--- /dev/null
+++ b/oap-server/server-admin/runtime-rule/src/test/java/org/apache/skywalking/oap/server/receiver/runtimerule/extension/DbOverrideRuntimeRuleResolverTest.java
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.receiver.runtimerule.extension;
+
+import java.lang.reflect.Method;
+import java.nio.charset.StandardCharsets;
+import org.apache.skywalking.oap.server.core.rule.ext.StaticRuleRegistry;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Covers the layer-override-forbidden guard {@link DbOverrideRuntimeRuleResolver}
+ * applies to runtime overrides of bundled MAL/LAL rules. The guard exists because
+ * bundled and runtime are separate ownership channels for layers; the runtime path
+ * must not overwrite the bundled channel.
+ */
+class DbOverrideRuntimeRuleResolverTest {
+
+ @BeforeEach
+ @AfterEach
+ void clearRegistry() throws Exception {
+ // StaticRuleRegistry is process-wide; isolate from sibling tests.
+ final Method m = StaticRuleRegistry.class.getDeclaredMethod("clear");
+ m.setAccessible(true);
+ m.invoke(StaticRuleRegistry.active());
+ }
+
+ @Test
+ void overrideWithoutLayerDefinitionsIsAllowed() {
+ StaticRuleRegistry.active().record(
+ "otel-rules", "no_layers_bundled",
+ "metricPrefix: foo\nmetricsRules:\n - name: x\n exp: a.sum(['s'])\n"
+ .getBytes(StandardCharsets.UTF_8));
+ final byte[] runtimeBody =
+ "metricPrefix: foo\nmetricsRules:\n - name: x\n exp: a.sum(['s'])\n"
+ .getBytes(StandardCharsets.UTF_8);
+ assertFalse(DbOverrideRuntimeRuleResolver.isLayerOverrideOnBundled(
+ "otel-rules", "no_layers_bundled", runtimeBody));
+ }
+
+ @Test
+ void overrideWithLayerDefinitionsOnDiskTwinIsForbidden() {
+ // Disk twin exists.
+ StaticRuleRegistry.active().record(
+ "otel-rules", "with_layers_bundled",
+ "metricPrefix: foo\nmetricsRules:\n - name: x\n exp: a.sum(['s'])\n"
+ .getBytes(StandardCharsets.UTF_8));
+ // Runtime override adds layerDefinitions — the case the resolver must drop.
+ final String override =
+ "metricPrefix: foo\n"
+ + "layerDefinitions:\n"
+ + " - name: TEST_RT_OVERRIDE_LAYER\n"
+ + " ordinal: 100700\n"
+ + "metricsRules:\n - name: x\n exp: a.sum(['s'])\n";
+ assertTrue(DbOverrideRuntimeRuleResolver.isLayerOverrideOnBundled(
+ "otel-rules", "with_layers_bundled",
+ override.getBytes(StandardCharsets.UTF_8)));
+ }
+
+ @Test
+ void pureRuntimeRuleWithLayerDefinitionsIsNotForbidden() {
+ // No disk twin recorded — pure runtime rule. layerDefinitions are allowed and
+ // go through the dynamic channel via RuleSync.runOnce. The guard returns false.
+ final String pureRuntime =
+ "metricPrefix: foo\n"
+ + "layerDefinitions:\n"
+ + " - name: TEST_PURE_RUNTIME_LAYER\n"
+ + " ordinal: 100701\n"
+ + "metricsRules:\n - name: x\n exp: a.sum(['s'])\n";
+ assertFalse(DbOverrideRuntimeRuleResolver.isLayerOverrideOnBundled(
+ "otel-rules", "no_disk_twin_pure_runtime",
+ pureRuntime.getBytes(StandardCharsets.UTF_8)));
+ }
+
+ @Test
+ void unknownCatalogReturnsFalse() {
+ // Catalog the guard doesn't know how to parse — returns false (apply will fail
+ // its own validation later if the content is malformed).
+ assertFalse(DbOverrideRuntimeRuleResolver.isLayerOverrideOnBundled(
+ "unknown-catalog", "x",
+ "anything".getBytes(StandardCharsets.UTF_8)));
+ }
+
+ @Test
+ void malformedYamlReturnsFalse() {
+ StaticRuleRegistry.active().record(
+ "otel-rules", "malformed_check",
+ "metricPrefix: foo\n".getBytes(StandardCharsets.UTF_8));
+ // Garbage bytes: the guard returns false rather than throwing. The apply path
+ // surfaces the YAML error with a clearer message; the override-forbidden guard
+ // is not the right place to flag parse failures.
+ assertFalse(DbOverrideRuntimeRuleResolver.isLayerOverrideOnBundled(
+ "otel-rules", "malformed_check",
+ "this: is: not: valid: yaml: at: all".getBytes(StandardCharsets.UTF_8)));
+ }
+}
diff --git a/oap-server/server-admin/runtime-rule/src/test/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/RuntimeLayerRegistryTest.java b/oap-server/server-admin/runtime-rule/src/test/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/RuntimeLayerRegistryTest.java
new file mode 100644
index 0000000000..2b595145a9
--- /dev/null
+++ b/oap-server/server-admin/runtime-rule/src/test/java/org/apache/skywalking/oap/server/receiver/runtimerule/layer/RuntimeLayerRegistryTest.java
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.receiver.runtimerule.layer;
+
+import java.util.Arrays;
+import java.util.Collections;
+import org.apache.skywalking.oap.server.core.analysis.Layer;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Exercises {@link RuntimeLayerRegistry}'s refcount semantics: net-new register/unregister
+ * on first/last claim, idempotent re-claim by the same rule, multi-rule sharing, and the
+ * apply/rollback round-trip used by the structural commit coordinator.
+ *
+ * <p>Uses ordinals in {@code [200_000, 200_999]} so collisions with sibling tests and any
+ * boot-time registrations cannot occur. Cleanup in {@link #cleanup} unregisters every
+ * dynamic layer it touched so JVM-shared state stays clean for other test classes.
+ */
+class RuntimeLayerRegistryTest {
+
+ private static final String RULE_A = "otel-rules/test-a";
+ private static final String RULE_B = "otel-rules/test-b";
+
+ private final RuntimeLayerRegistry registry = new RuntimeLayerRegistry();
+
+ @AfterEach
+ void cleanup() {
+ // Drop every claim, which cascades into Layer.unregisterDynamic for any net-zero
+ // layer. Defensive: walk a snapshot so we don't mutate the map mid-iteration.
+ for (final String rule : Arrays.asList(RULE_A, RULE_B, "otel-rules/test-c")) {
+ registry.removeRule(rule);
+ }
+ }
+
+ @Test
+ void applyRegistersNewLayerAndCountsClaim() {
+ final LayerClaim claim = new LayerClaim("DYN_REGISTRY_A", 200_000, true);
+ final AppliedClaims applied = registry.apply(RULE_A, Collections.singletonList(claim));
+
+ assertTrue(applied.getNewlyRegistered().contains("DYN_REGISTRY_A"));
+ assertEquals(Collections.singleton("DYN_REGISTRY_A"), registry.claimsOf(RULE_A));
+ assertEquals(200_000, Layer.nameOf("DYN_REGISTRY_A").value());
+ }
+
+ @Test
+ void secondClaimOnSameLayerIsIdempotent() {
+ final LayerClaim claim = new LayerClaim("DYN_REGISTRY_SHARED", 200_001, true);
+ final AppliedClaims firstApplied = registry.apply(RULE_A, Collections.singletonList(claim));
+ final AppliedClaims secondApplied = registry.apply(RULE_B, Collections.singletonList(claim));
+
+ // First apply registers the layer; second only adds B as a claimant.
+ assertTrue(firstApplied.getNewlyRegistered().contains("DYN_REGISTRY_SHARED"));
+ assertTrue(secondApplied.getNewlyRegistered().isEmpty());
+
+ // Both A and B are claimants.
+ assertEquals(2, registry.snapshot().get("DYN_REGISTRY_SHARED").size());
+ }
+
+ @Test
+ void unregisterDropsLayerWhenLastClaimGone() {
+ final LayerClaim claim = new LayerClaim("DYN_REGISTRY_DROP", 200_002, true);
+ registry.apply(RULE_A, Collections.singletonList(claim));
+ registry.apply(RULE_B, Collections.singletonList(claim));
+
+ // Removing A keeps the layer alive (B still claims it).
+ registry.removeRule(RULE_A);
+ assertEquals(200_002, Layer.nameOf("DYN_REGISTRY_DROP").value());
+
+ // Removing B drops the last claim — layer goes away.
+ registry.removeRule(RULE_B);
+ assertSame(Layer.UNDEFINED, Layer.nameOf("DYN_REGISTRY_DROP"));
+ }
+
+ @Test
+ void updateReplacesPriorClaimsAndUnregistersOrphans() {
+ final LayerClaim original = new LayerClaim("DYN_REGISTRY_ORPHAN", 200_003, true);
+ final LayerClaim replacement = new LayerClaim("DYN_REGISTRY_KEEP", 200_004, true);
+
+ registry.apply(RULE_A, Collections.singletonList(original));
+ assertEquals(200_003, Layer.nameOf("DYN_REGISTRY_ORPHAN").value());
+
+ registry.apply(RULE_A, Collections.singletonList(replacement));
+
+ // The orphaned layer is gone (no other rule claims it).
+ assertSame(Layer.UNDEFINED, Layer.nameOf("DYN_REGISTRY_ORPHAN"));
+ // The replacement is now registered and claimed by RULE_A.
+ assertEquals(200_004, Layer.nameOf("DYN_REGISTRY_KEEP").value());
+ assertEquals(Collections.singleton("DYN_REGISTRY_KEEP"), registry.claimsOf(RULE_A));
+ }
+
+ @Test
+ void rollbackRestoresPriorStateAfterAppliedUpdate() {
+ final LayerClaim original = new LayerClaim("DYN_REGISTRY_ROLLBACK_PRIOR", 200_005, true);
+ final LayerClaim replacement = new LayerClaim("DYN_REGISTRY_ROLLBACK_NEW", 200_006, true);
+
+ registry.apply(RULE_A, Collections.singletonList(original));
+ final AppliedClaims applied = registry.apply(RULE_A, Collections.singletonList(replacement));
+
+ // Confirm intermediate state.
+ assertSame(Layer.UNDEFINED, Layer.nameOf("DYN_REGISTRY_ROLLBACK_PRIOR"));
+ assertEquals(200_006, Layer.nameOf("DYN_REGISTRY_ROLLBACK_NEW").value());
+
+ registry.rollback(applied);
+
+ // Prior layer restored; replacement unregistered.
+ assertEquals(200_005, Layer.nameOf("DYN_REGISTRY_ROLLBACK_PRIOR").value());
+ assertSame(Layer.UNDEFINED, Layer.nameOf("DYN_REGISTRY_ROLLBACK_NEW"));
+ assertEquals(Collections.singleton("DYN_REGISTRY_ROLLBACK_PRIOR"),
+ registry.claimsOf(RULE_A));
+ }
+
+ @Test
+ void selfUpdateWithDifferentOrdinalIsPermitted() {
+ // Rule edits its OWN previously-declared layer — conflict checker treats the
+ // prior claim as already-removed, so a triple change is allowed.
+ final LayerClaim before = new LayerClaim("DYN_REGISTRY_SELF_EDIT", 200_007, true);
+ final LayerClaim after = new LayerClaim("DYN_REGISTRY_SELF_EDIT", 200_008, false);
+
+ registry.apply(RULE_A, Collections.singletonList(before));
+ registry.apply(RULE_A, Collections.singletonList(after));
+
+ final Layer current = Layer.nameOf("DYN_REGISTRY_SELF_EDIT");
+ assertEquals(200_008, current.value());
+ assertEquals(false, current.isNormal());
+ }
+
+ @Test
+ void conflictAcrossRulesRaisesLayerNameConflict() {
+ registry.apply(RULE_A, Collections.singletonList(
+ new LayerClaim("DYN_REGISTRY_CROSS_RULE", 200_009, true)));
+
+ // Rule B declares same name but different ordinal — must be rejected.
+ final LayerConflictException ex = assertThrows(
+ LayerConflictException.class,
+ () -> registry.apply(RULE_B, Collections.singletonList(
+ new LayerClaim("DYN_REGISTRY_CROSS_RULE", 200_010, true))));
+
+ assertEquals(LayerConflictException.Status.LAYER_NAME_CONFLICT, ex.getStatus());
+ assertEquals("layer_name_conflict", ex.applyStatus());
+ assertTrue(ex.getMessage().contains("DYN_REGISTRY_CROSS_RULE"));
+ assertTrue(ex.getMessage().contains(RULE_A), "message must name the source rule");
+ assertTrue(ex.getMessage().contains(RULE_B), "message must name the offender");
+ }
+
+ @Test
+ void conflictRaisesOrdinalCollisionForDifferentNameSameOrdinal() {
+ registry.apply(RULE_A, Collections.singletonList(
+ new LayerClaim("DYN_REGISTRY_ORD_A", 200_011, true)));
+
+ final LayerConflictException ex = assertThrows(
+ LayerConflictException.class,
+ () -> registry.apply(RULE_B, Collections.singletonList(
+ new LayerClaim("DYN_REGISTRY_ORD_B", 200_011, true))));
+
+ assertEquals(LayerConflictException.Status.LAYER_ORDINAL_COLLISION, ex.getStatus());
+ assertEquals("layer_ordinal_collision", ex.applyStatus());
+ assertTrue(ex.getMessage().contains("200011") || ex.getMessage().contains("200_011"));
+ }
+
+ @Test
+ void conflictRaisesOrdinalOutOfRangeBelowFloor() {
+ final LayerConflictException ex = assertThrows(
+ LayerConflictException.class,
+ () -> registry.apply(RULE_A, Collections.singletonList(
+ new LayerClaim("DYN_REGISTRY_LOW", 99_999, true))));
+
+ assertEquals(LayerConflictException.Status.LAYER_ORDINAL_OUT_OF_RANGE, ex.getStatus());
+ assertTrue(ex.getMessage().contains("99999"));
+ assertTrue(ex.getMessage().contains("100000"));
+ }
+
+ @Test
+ void conflictRaisesNameInvalidForBadShape() {
+ final LayerConflictException ex = assertThrows(
+ LayerConflictException.class,
+ () -> registry.apply(RULE_A, Collections.singletonList(
+ new LayerClaim("bad-name", 200_012, true))));
+
+ assertEquals(LayerConflictException.Status.LAYER_NAME_INVALID, ex.getStatus());
+ assertTrue(ex.getMessage().contains("bad-name"));
+ }
+
+ @Test
+ void conflictAgainstBundledLayerLabelsSourceClearly() {
+ // MESH is the built-in layer with ordinal 1 — declaring it with a runtime
+ // ordinal yields a name conflict; the source must be labelled "built-in or
+ // boot-time extension" so the operator knows it can't be aligned at all.
+ final LayerConflictException ex = assertThrows(
+ LayerConflictException.class,
+ () -> registry.apply(RULE_A, Collections.singletonList(
+ new LayerClaim("MESH", 200_013, true))));
+
+ assertEquals(LayerConflictException.Status.LAYER_NAME_CONFLICT, ex.getStatus());
+ assertTrue(ex.getMessage().contains("MESH"));
+ assertTrue(ex.getMessage().contains("built-in"),
+ "message must label the source as built-in: " + ex.getMessage());
+ }
+
+ @Test
+ void inBatchDuplicateNameWithDifferentTriplesIsRejected() {
+ // A single batch declaring the same name twice with different triples can never
+ // be satisfied — must be caught BEFORE any mutation lands so apply stays atomic.
+ final LayerConflictException ex = assertThrows(
+ LayerConflictException.class,
+ () -> registry.apply(RULE_A, Arrays.asList(
+ new LayerClaim("DYN_BATCH_DUP_NAME", 200_020, true),
+ new LayerClaim("DYN_BATCH_DUP_NAME", 200_021, true))));
+ assertEquals(LayerConflictException.Status.LAYER_NAME_CONFLICT, ex.getStatus());
+ // Neither variant should have been registered — the registry must look unchanged.
+ assertSame(Layer.UNDEFINED,
+ Layer.nameOf("DYN_BATCH_DUP_NAME"));
+ assertTrue(registry.claimsOf(RULE_A).isEmpty());
+ }
+
+ @Test
+ void inBatchDuplicateOrdinalUnderDifferentNamesIsRejected() {
+ final LayerConflictException ex = assertThrows(
+ LayerConflictException.class,
+ () -> registry.apply(RULE_A, Arrays.asList(
+ new LayerClaim("DYN_BATCH_ORD_X", 200_022, true),
+ new LayerClaim("DYN_BATCH_ORD_Y", 200_022, true))));
+ assertEquals(LayerConflictException.Status.LAYER_ORDINAL_COLLISION, ex.getStatus());
+ // Neither name registered.
+ assertSame(Layer.UNDEFINED,
+ Layer.nameOf("DYN_BATCH_ORD_X"));
+ assertSame(Layer.UNDEFINED,
+ Layer.nameOf("DYN_BATCH_ORD_Y"));
+ }
+
+ @Test
+ void inBatchSameTripleRepeatedIsAccepted() {
+ // Idempotent duplicate within a batch (operator copy-paste, or a generator that
+ // emits the same entry twice) is harmless and must not be rejected.
+ registry.apply(RULE_A, Arrays.asList(
+ new LayerClaim("DYN_BATCH_IDEMPOTENT", 200_023, true),
+ new LayerClaim("DYN_BATCH_IDEMPOTENT", 200_023, true)));
+ assertEquals(200_023,
+ Layer
+ .nameOf("DYN_BATCH_IDEMPOTENT").value());
+ }
+
+ @Test
+ void selfUpdateSwappingOrdinalsAcrossTwoLayersIsAtomic() {
+ // Rule R declares LAYER_X@200_030 and LAYER_Y@200_031. Operator pushes a swap:
+ // LAYER_X@200_031, LAYER_Y@200_030. validate must permit it (each is a self-claim
+ // freeing the other's ordinal), apply must execute the swap without intermediate
+ // ordinal collision throwing — that's the bug finding 2 surfaced for the per-claim
+ // loop. The two-pass apply unregisters all prior claims first, then registers all
+ // new triples, so the freed slots are available for the swap.
+ registry.apply(RULE_A, Arrays.asList(
+ new LayerClaim("DYN_SWAP_X", 200_030, true),
+ new LayerClaim("DYN_SWAP_Y", 200_031, true)));
+ registry.apply(RULE_A, Arrays.asList(
+ new LayerClaim("DYN_SWAP_X", 200_031, true),
+ new LayerClaim("DYN_SWAP_Y", 200_030, true)));
+
+ assertEquals(200_031,
+ Layer
+ .nameOf("DYN_SWAP_X").value());
+ assertEquals(200_030,
+ Layer
+ .nameOf("DYN_SWAP_Y").value());
+ }
+
+ @Test
+ void softClaimAgainstBundledLayerSucceedsButCannotRemove() {
+ // Pre-register a layer through the bundled channel (Layer.register) at an
+ // ordinal in the runtime range. This simulates a boot-time external entry
+ // (layer-extensions.yml / bundled MAL/LAL) that — against the recommended
+ // tier convention — happens to live at >=100_000. A runtime rule claiming the
+ // SAME triple gets a soft claim: refcount tracks the rule, Layer stays bundled,
+ // unregisterDynamic refuses on rule delete.
+ Layer.register("DYN_REG_SOFT_CLAIM_TARGET", 250_010, true);
+ try {
+ registry.apply(RULE_A, Collections.singletonList(
+ new LayerClaim("DYN_REG_SOFT_CLAIM_TARGET", 250_010, true)));
+
+ assertTrue(registry.snapshot().get("DYN_REG_SOFT_CLAIM_TARGET").contains(RULE_A));
+ assertFalse(Layer.isDynamic("DYN_REG_SOFT_CLAIM_TARGET"),
+ "soft claim does not flip the channel from bundled to dynamic");
+ assertEquals(250_010, Layer.nameOf("DYN_REG_SOFT_CLAIM_TARGET").value());
+
+ registry.removeRule(RULE_A);
+
+ assertEquals(250_010, Layer.nameOf("DYN_REG_SOFT_CLAIM_TARGET").value(),
+ "removing the soft-claimant rule must NOT unregister the bundled layer");
+ assertFalse(registry.snapshot().containsKey("DYN_REG_SOFT_CLAIM_TARGET"));
+ } finally {
+ // Cannot unregister via the dynamic path (it's bundled). Layer stays in the
+ // JVM-wide registry but the unique name prevents sibling-test interference.
+ }
+ }
+
+ @Test
+ void selfUpdateReusingMultiClaimantOrdinalIsRejectedStructured() {
+ // Rules A + B both claim OLD@200_090. A then tries to update to NEW@200_090 —
+ // reusing the ordinal of its own prior layer under a new name. Without the
+ // multi-claimant check, validate would pass (A held OLD priorly), apply's Pass-1
+ // would only drop A's claim from OLD (B still holds it), and registerDynamic
+ // would throw an unstructured ordinal collision deep in apply. The structured
+ // layer_ordinal_collision must surface from validate before any mutation lands.
+ final LayerClaim shared = new LayerClaim("DYN_MULTI_CLAIM_OLD", 200_090, true);
+ registry.apply(RULE_A, Collections.singletonList(shared));
+ registry.apply(RULE_B, Collections.singletonList(shared));
+
+ final LayerConflictException ex = assertThrows(
+ LayerConflictException.class,
+ () -> registry.apply(RULE_A, Collections.singletonList(
+ new LayerClaim("DYN_MULTI_CLAIM_NEW", 200_090, true))));
+ assertEquals(LayerConflictException.Status.LAYER_ORDINAL_COLLISION, ex.getStatus());
+ assertTrue(ex.getMessage().contains(RULE_B),
+ "message must name the rule still holding the prior layer");
+ // State unchanged: OLD still owned by A + B, no NEW.
+ assertSame(Layer.UNDEFINED, Layer.nameOf("DYN_MULTI_CLAIM_NEW"));
+ assertEquals(200_090, Layer.nameOf("DYN_MULTI_CLAIM_OLD").value());
+ }
+
+ @Test
+ void reactivationAfterInactiveSurfacesConflictAgainstNewClaimant() {
+ // Rule A claims a dynamic layer. A is then "inactivated" (its claims removed
+ // from the registry). While A is inactive, rule B starts claiming the same
+ // name at a different ordinal. When the operator reactivates A, the apply
+ // must surface the conflict so the operator can edit or drop A.
+ registry.apply(RULE_A, Collections.singletonList(
+ new LayerClaim("DYN_REACTIVATE_PROBE", 200_080, true)));
+ // Inactivate A.
+ registry.removeRule(RULE_A);
+ assertSame(Layer.UNDEFINED, Layer.nameOf("DYN_REACTIVATE_PROBE"));
+
+ // Meanwhile, B claims the same name at a different ordinal.
+ registry.apply(RULE_B, Collections.singletonList(
+ new LayerClaim("DYN_REACTIVATE_PROBE", 200_081, true)));
+
+ // Operator pushes A again (reactivation). The original triple no longer fits.
+ final LayerConflictException ex = assertThrows(
+ LayerConflictException.class,
+ () -> registry.apply(RULE_A, Collections.singletonList(
+ new LayerClaim("DYN_REACTIVATE_PROBE", 200_080, true))));
+ assertEquals(LayerConflictException.Status.LAYER_NAME_CONFLICT, ex.getStatus());
+ assertTrue(ex.getMessage().contains(RULE_B),
+ "message must name the currently-active claimant so operator knows what to align with");
+ }
+
+ @Test
+ void noOpRedeclareDoesNotPerturbLiveLayer() {
+ // Re-applying the SAME triple by the SAME rule should be a no-op at the Layer
+ // level (no unregister + re-register churn that briefly leaves the layer
+ // unresolvable). Confirmed by re-checking the identity of the Layer instance
+ // before and after the second apply.
+ registry.apply(RULE_A, Collections.singletonList(
+ new LayerClaim("DYN_STABLE_REAPPLY", 200_040, true)));
+ final Layer before =
+ Layer.nameOf("DYN_STABLE_REAPPLY");
+ registry.apply(RULE_A, Collections.singletonList(
+ new LayerClaim("DYN_STABLE_REAPPLY", 200_040, true)));
+ final Layer after =
+ Layer.nameOf("DYN_STABLE_REAPPLY");
+ assertSame(before, after, "same-triple re-apply must not churn the Layer instance");
+ }
+}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java
index 0d7e1a8354..bb702d5c1a 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java
@@ -20,6 +20,7 @@ package org.apache.skywalking.oap.server.core.analysis;
import java.util.Comparator;
import java.util.Map;
+import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
@@ -35,35 +36,69 @@ import org.apache.skywalking.oap.server.core.UnexpectedException;
* {@link #valueOf(String)}) or by ordinal ({@link #valueOf(int)}); {@link #values()} returns
* every registered layer sorted by ordinal.
*
- * <p>Layers are persisted by ordinal (int). Built-in ordinals 0–49 are frozen forever and
- * must never be reused. Ordinals 50–999 are reserved by convention for future built-ins;
- * external extensions are <strong>recommended</strong> to start at {@code 1000} to avoid
- * colliding with future built-ins on OAP upgrade. The recommendation is informational and
- * not enforced — any non-colliding ordinal is accepted — but a future built-in landing on
- * the same ordinal as an extension layer will cause OAP boot to fail loudly via the
- * ordinal uniqueness check, which is the upgrade-time detection mechanism.
+ * <p>Layers are persisted by ordinal (int). The ordinal space is partitioned by tier so
+ * each registration channel has a non-overlapping range:
+ * <ul>
+ * <li>{@code 0 – 9_999} — built-in {@code Layer.*} constants. Currently {@code 0..49} in
+ * use; the rest of the range is reserved for future built-ins. Ordinals already
+ * persisted in storage are frozen forever and must never be reused.</li>
+ * <li>{@code 10_000 – 99_999} — boot-time external layers: {@code layer-extensions.yml},
+ * {@code LayerExtension} SPI, and bundled MAL/LAL {@code layerDefinitions:} blocks.
+ * Registered through {@link #register} before {@link #seal()}.</li>
+ * <li>{@code 100_000 – Integer.MAX_VALUE} — runtime DSL dynamic layers introduced by
+ * MAL/LAL hot-update {@code layerDefinitions:} blocks. Registered through
+ * {@link #registerDynamic} after {@link #seal()} and removable through
+ * {@link #unregisterDynamic} when the last declaring rule is removed.</li>
+ * </ul>
+ * The tier boundaries are enforced for {@link #registerDynamic} / {@link #unregisterDynamic}
+ * (ordinal must be {@code >= 100_000}). {@link #register} accepts any non-colliding ordinal
+ * for backward compatibility, but operator-managed extensions <strong>should</strong> use
+ * the {@code 10_000+} tier to leave the {@code 100_000+} space for runtime DSL.
*
- * <p>Registration paths (all funnel through {@link #register}):
+ * <p>Registration paths (all funnel through {@link #register} or {@link #registerDynamic}):
* <ol>
* <li>Operator yaml — {@code layer-extensions.yml} on the OAP classpath/config dir.</li>
* <li>Java SPI — {@code LayerExtension} discovered via {@code ServiceLoader}.</li>
- * <li>DSL inline — MAL/LAL rule files declaring a top-level {@code layerDefinitions:}
- * block; the DSL loader funnels each entry through this method before compiling.</li>
+ * <li>DSL inline (boot) — MAL/LAL rule files declaring a top-level {@code layerDefinitions:}
+ * block; the DSL loader funnels each entry through {@link #register} before compiling.</li>
+ * <li>DSL inline (runtime) — runtime-rule MAL/LAL hot-update files declaring
+ * {@code layerDefinitions:}; funneled through {@link #registerDynamic}. The runtime-rule
+ * module's {@code RuntimeLayerRegistry} refcounts these per declaring rule file and
+ * invokes {@link #unregisterDynamic} when the last claim drops.</li>
* </ol>
* The registry is sealed at the start of {@code CoreModuleProvider.notifyAfterCompleted()};
- * any registration attempt after that throws.
+ * after seal, only {@link #registerDynamic} and {@link #unregisterDynamic} can mutate the
+ * registry.
*/
public final class Layer {
private static final Pattern NAME_PATTERN = Pattern.compile("[A-Z][A-Z0-9_]*");
+ /**
+ * Lowest ordinal allowed for runtime DSL dynamic layers. Ordinals below this value are
+ * reserved for built-in {@code Layer.*} constants ({@code 0–9_999}) and boot-time
+ * external extensions ({@code 10_000–99_999}). See class javadoc for the full tier
+ * convention.
+ */
+ public static final int RUNTIME_DYNAMIC_MIN_ORDINAL = 100_000;
+
private static final Map<Integer, Layer> BY_VALUE = new ConcurrentHashMap<>();
private static final Map<String, Layer> BY_NAME = new ConcurrentHashMap<>();
+ /**
+ * Explicit ownership marker: every name added through {@link #registerDynamic} lands
+ * here, and {@link #unregisterDynamic} consults this set to refuse removing layers
+ * that came through the boot-time {@link #register} channel. Distinguishing by ordinal
+ * range alone (the original heuristic) was unsafe — operator yaml or bundled MAL/LAL
+ * can land in the {@code >=100_000} range and would have been wrongly considered
+ * removable. Adding here is idempotent on re-registration.
+ */
+ private static final Set<String> DYNAMIC_NAMES = ConcurrentHashMap.newKeySet();
private static volatile boolean SEALED = false;
/**
- * Snapshot of all registered layers sorted by ordinal, frozen at {@link #seal()}.
- * Before seal, {@link #values()} computes on the fly so newly registered layers are
- * visible; after seal, {@link #values()} returns a clone of this cached array.
+ * Snapshot of all registered layers sorted by ordinal. Populated once at {@link #seal()};
+ * after seal, {@link #registerDynamic} / {@link #unregisterDynamic} rebuild this array
+ * so {@link #values()} reflects post-seal mutations. Before seal, {@link #values()}
+ * throws because the registry may still grow via boot-time {@link #register} calls.
*/
private static volatile Layer[] CACHED_VALUES = null;
@@ -341,16 +376,133 @@ public final class Layer {
}
/**
- * Closes the registry. Subsequent {@link #register} calls throw. Called by
- * {@code CoreModuleProvider.notifyAfterCompleted()} after every module's prepare/start
- * has run, so MAL/LAL/SPI/yaml all had their full window. Idempotent.
+ * Closes the registry. Subsequent {@link #register} calls throw; only
+ * {@link #registerDynamic} / {@link #unregisterDynamic} can mutate the registry after
+ * seal. Called by {@code CoreModuleProvider.notifyAfterCompleted()} after every
+ * module's prepare/start has run, so MAL/LAL/SPI/yaml all had their full window.
+ * Idempotent.
*/
public static synchronized void seal() {
+ rebuildCachedValues();
+ SEALED = true;
+ }
+
+ /**
+ * Post-seal registration entry point for runtime DSL hot-update. Mirrors
+ * {@link #register} (same name regex, name uniqueness, ordinal uniqueness, idempotent
+ * on identical re-registration) but bypasses the seal check so the runtime-rule module
+ * can introduce layers during a hot-update apply.
+ *
+ * <p>Enforces {@code ordinal >= }{@link #RUNTIME_DYNAMIC_MIN_ORDINAL} so runtime layers
+ * cannot collide with built-in ordinals or boot-time external extensions. The caller
+ * (runtime-rule module's {@code RuntimeLayerRegistry}) is responsible for refcounting
+ * declarations and invoking {@link #unregisterDynamic} when the last declaring rule is
+ * removed.
+ *
+ * <p>Rebuilds the {@link #values()} snapshot so post-seal additions are visible to
+ * subsequent iterators.
+ *
+ * @throws IllegalArgumentException if name shape is invalid or {@code value < }
+ * {@link #RUNTIME_DYNAMIC_MIN_ORDINAL}
+ * @throws IllegalStateException on name/ordinal conflict with an existing layer
+ */
+ public static synchronized Layer registerDynamic(final String name, final int value, final boolean isNormal) {
+ if (value < RUNTIME_DYNAMIC_MIN_ORDINAL) {
+ throw new IllegalArgumentException(
+ "Runtime dynamic layer ordinal must be >= " + RUNTIME_DYNAMIC_MIN_ORDINAL
+ + " (received " + value + " for layer '" + name + "'). "
+ + "Reserved ranges: 0-9999 = built-in Layer constants, "
+ + "10_000-99_999 = layer-extensions.yml + bundled MAL/LAL layerDefinitions.");
+ }
+ if (name == null || !NAME_PATTERN.matcher(name).matches()) {
+ throw new IllegalArgumentException(
+ "Layer name must match [A-Z][A-Z0-9_]*: " + name);
+ }
+ final Layer existingByName = BY_NAME.get(name);
+ if (existingByName != null) {
+ if (existingByName.value == value && existingByName.isNormal == isNormal) {
+ return existingByName;
+ }
+ throw new IllegalStateException(
+ "Layer name conflict: " + name + " already registered as ordinal=" + existingByName.value
+ + ", normal=" + existingByName.isNormal
+ + "; refused re-registration as ordinal=" + value + ", normal=" + isNormal);
+ }
+ final Layer existingByValue = BY_VALUE.get(value);
+ if (existingByValue != null) {
+ throw new IllegalStateException(
+ "Layer ordinal conflict at " + value
+ + ": existing=" + existingByValue.name + ", new=" + name);
+ }
+ final Layer layer = new Layer(name, value, isNormal);
+ BY_VALUE.put(value, layer);
+ BY_NAME.put(name, layer);
+ DYNAMIC_NAMES.add(name);
+ rebuildCachedValues();
+ return layer;
+ }
+
+ /**
+ * Remove a layer added via {@link #registerDynamic}. No-op if the name is not currently
+ * registered. Throws if the named layer was NOT registered through the dynamic
+ * channel — built-in {@code Layer.*} constants, {@code layer-extensions.yml} entries,
+ * and bundled MAL/LAL {@code layerDefinitions:} blocks all funnel through
+ * {@link #register} and are non-removable because their ordinals are baked into
+ * persisted entity primary keys (e.g. {@code ServiceTraffic.id}).
+ *
+ * <p>Ownership is tracked explicitly through {@link #DYNAMIC_NAMES}, not inferred from
+ * the ordinal range — operators are free to put boot-time external layers anywhere in
+ * the registry's accepted range, so an ordinal-only check would wrongly accept removal
+ * of a boot-time layer that happens to live in the {@code >=100_000} tier.
+ *
+ * <p>Rebuilds the {@link #values()} snapshot.
+ *
+ * @throws IllegalStateException if the named layer was not registered via
+ * {@link #registerDynamic}
+ */
+ public static synchronized void unregisterDynamic(final String name) {
+ if (name == null) {
+ return;
+ }
+ final Layer existing = BY_NAME.get(name);
+ if (existing == null) {
+ return;
+ }
+ if (!DYNAMIC_NAMES.contains(name)) {
+ throw new IllegalStateException(
+ "Refusing to unregister non-dynamic layer " + name + " (ordinal="
+ + existing.value + "). Only layers registered through "
+ + "registerDynamic are removable; this one came through register "
+ + "(built-in Layer constant, layer-extensions.yml, or bundled "
+ + "MAL/LAL layerDefinitions).");
+ }
+ BY_NAME.remove(name);
+ BY_VALUE.remove(existing.value);
+ DYNAMIC_NAMES.remove(name);
+ rebuildCachedValues();
+ }
+
+ /**
+ * True when the named layer was registered through {@link #registerDynamic} — i.e.
+ * owned by the runtime-rule module's {@code RuntimeLayerRegistry} rather than by the
+ * boot-time {@link #register} channel. Used by the boot-time MAL/LAL static loaders
+ * to skip {@code layerDefinitions:} entries the runtime path has already handled.
+ */
+ public static boolean isDynamic(final String name) {
+ return name != null && DYNAMIC_NAMES.contains(name);
+ }
+
+ /**
+ * Recompute {@link #CACHED_VALUES} from {@link #BY_VALUE}, sorted by ordinal. Called on
+ * {@link #seal()} and on every post-seal mutation through {@link #registerDynamic} /
+ * {@link #unregisterDynamic}. Pre-seal callers iterating {@link #values()} still hit
+ * the "not yet sealed" guard because {@link #SEALED} flips only inside {@link #seal()}.
+ */
+ private static void rebuildCachedValues() {
CACHED_VALUES = BY_VALUE.values()
.stream()
.sorted(Comparator.comparingInt(Layer::value))
.toArray(Layer[]::new);
- SEALED = true;
}
public static Layer valueOf(final int value) {
@@ -382,6 +534,17 @@ public final class Layer {
return layer;
}
+ /**
+ * Lenient ordinal lookup — returns {@code null} when no layer is registered at the
+ * given ordinal. Intended for callers that need to probe the registry without paying
+ * the cost (or noise) of catching {@link UnexpectedException} thrown by
+ * {@link #valueOf(int)}. Used by the runtime-rule conflict checker when validating
+ * dynamic-layer declarations before any registration is attempted.
+ */
+ public static Layer peekByValue(final int value) {
+ return BY_VALUE.get(value);
+ }
+
/**
* Snapshot of all registered layers, sorted by ordinal value. Returns a fresh array each
* call to preserve enum-style mutation safety; the underlying snapshot is computed once
@@ -394,7 +557,7 @@ public final class Layer {
* or defer the iteration to a post-boot phase.
*/
public static Layer[] values() {
- if (CACHED_VALUES == null) {
+ if (!SEALED) {
throw new IllegalStateException(
"Layer.values() called before the registry was sealed. "
+ "External layers may still register during module prepare()/start(); "
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/rule/ext/RuleSetMerger.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/rule/ext/RuleSetMerger.java
index d22240174c..9478dcee3c 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/rule/ext/RuleSetMerger.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/rule/ext/RuleSetMerger.java
@@ -148,7 +148,14 @@ public final class RuleSetMerger {
resolver.getClass().getName(), catalog, name);
return;
}
- out.put(name, res.getContent());
+ // Only override existing disk entries. Pure-runtime rules (no disk
+ // twin) are applied by RuleSync.runOnce post-seal via the dynamic
+ // channel; injecting them into the static loader here would
+ // register their layerDefinitions through Layer.register and
+ // permanently lose operator-removable ownership.
+ if (out.containsKey(name)) {
+ out.put(name, res.getContent());
+ }
break;
case INACTIVE:
out.remove(name);
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/rule/ext/RuntimeRuleOverrideResolver.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/rule/ext/RuntimeRuleOverrideResolver.java
index c3c3236724..3f0c341123 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/rule/ext/RuntimeRuleOverrideResolver.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/rule/ext/RuntimeRuleOverrideResolver.java
@@ -36,7 +36,11 @@ import org.apache.skywalking.oap.server.library.module.ModuleManager;
* {@code (name -> bytes)} map per catalog, with priority deciding ties:
* <ul>
* <li>Lower {@link #priority} resolvers are applied first; higher priority overwrites.</li>
- * <li>{@link Decision#ACTIVE} substitutes the resolver's content into the merged set.</li>
+ * <li>{@link Decision#ACTIVE} substitutes the resolver's content into the merged set
+ * <strong>only when the key already exists on disk</strong>. Resolver-only entries
+ * (no disk twin) are dropped — they're handled by the runtime-rule reconciler post-seal
+ * through {@code RuntimeLayerRegistry} so layerDefinitions go through the dynamic
+ * channel and remain operator-removable. The merger drops them silently.</li>
* <li>{@link Decision#INACTIVE} removes the entry from the merged set — even if the disk
* file or a lower-priority resolver had content for that key.</li>
* <li>A resolver omits a key from {@link #loadAll} when it has no opinion; the next higher
@@ -48,11 +52,13 @@ import org.apache.skywalking.oap.server.library.module.ModuleManager;
* Resolver A (priority 100, runtime-rule DB):
* "vm" => Resolution(ACTIVE, bytes-from-DB)
* "noisy-rule" => Resolution(INACTIVE, null)
+ * "new-rule" => Resolution(ACTIVE, bytes-from-DB) // no disk twin
*
* Result for catalog "otel-rules":
* - if disk has "vm.yaml": merged["vm"] = bytes-from-DB
* - if disk has "noisy-rule.yaml": merged drops "noisy-rule" entirely
- * - if disk lacks "new-rule.yaml": merged["new-rule"] = bytes-from-DB (DB-only rule)
+ * - if disk lacks "new-rule.yaml": merged does NOT contain "new-rule" (pure runtime —
+ * applied by RuleSync post-seal via the dynamic channel)
* </pre>
*/
public interface RuntimeRuleOverrideResolver {
diff --git a/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/analysis/LayerDynamicRegisterTest.java b/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/analysis/LayerDynamicRegisterTest.java
new file mode 100644
index 0000000000..8a238e650e
--- /dev/null
+++ b/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/analysis/LayerDynamicRegisterTest.java
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.analysis;
+
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Validates {@link Layer#registerDynamic(String, int, boolean)} and
+ * {@link Layer#unregisterDynamic(String)}: the post-seal runtime-DSL registration channel
+ * introduced for hot-update of MAL/LAL rules with new {@code layerDefinitions:} entries.
+ *
+ * <p>Tests assert state via {@link Layer#nameOf(String)} / {@link Layer#valueOf(int)} so the
+ * registry need not be sealed — sealing would taint sibling tests that share the JVM (the
+ * registry is a process-wide singleton). Visibility through {@link Layer#values()} is
+ * covered in the runtime-dynamic-layers L2 integration test, which runs in its own forked
+ * OAP process.
+ *
+ * <p>Ordinals used here live in the runtime tier ({@code >= 100_000}) so collisions with
+ * built-in constants, sibling tests' yaml fixtures (ordinals 1150-1199), and the
+ * conventional boot-time external range (10_000-99_999) are impossible by construction.
+ */
+class LayerDynamicRegisterTest {
+
+ @Test
+ void registerDynamicRejectsOrdinalBelowFloor() {
+ // 99_999 is one below the runtime floor — must be rejected with the operator-actionable
+ // message naming the actual ordinal and reserved ranges.
+ final IllegalArgumentException ex = assertThrows(
+ IllegalArgumentException.class,
+ () -> Layer.registerDynamic("DYN_FLOOR_REJECT", 99_999, true));
+ // Message must surface the ordinal so the operator can correlate with their yaml.
+ assert ex.getMessage().contains("99999") || ex.getMessage().contains("99_999");
+ assert ex.getMessage().contains("100000") || ex.getMessage().contains("100_000");
+ }
+
+ @Test
+ void registerDynamicAtFloorSucceeds() {
+ final Layer layer = Layer.registerDynamic("DYN_AT_FLOOR", 100_000, true);
+ assertEquals("DYN_AT_FLOOR", layer.name());
+ assertEquals(100_000, layer.value());
+ assertEquals(true, layer.isNormal());
+ // Round-trip lookup.
+ assertSame(layer, Layer.nameOf("DYN_AT_FLOOR"));
+ assertSame(layer, Layer.valueOf(100_000));
+ }
+
+ @Test
+ void registerDynamicIsIdempotentOnIdenticalTriple() {
+ final Layer first = Layer.registerDynamic("DYN_IDEMPOTENT", 100_001, true);
+ final Layer second = Layer.registerDynamic("DYN_IDEMPOTENT", 100_001, true);
+ assertSame(first, second);
+ }
+
+ @Test
+ void registerDynamicRejectsNameConflict() {
+ Layer.registerDynamic("DYN_NAME_CONFLICT", 100_002, true);
+ // Same name, different ordinal.
+ assertThrows(IllegalStateException.class,
+ () -> Layer.registerDynamic("DYN_NAME_CONFLICT", 100_003, true));
+ // Same name, different normal flag.
+ assertThrows(IllegalStateException.class,
+ () -> Layer.registerDynamic("DYN_NAME_CONFLICT", 100_002, false));
+ }
+
+ @Test
+ void registerDynamicRejectsOrdinalCollisionWithDifferentName() {
+ Layer.registerDynamic("DYN_ORDINAL_FIRST", 100_004, true);
+ assertThrows(IllegalStateException.class,
+ () -> Layer.registerDynamic("DYN_ORDINAL_SECOND", 100_004, true));
+ }
+
+ @Test
+ void registerDynamicRejectsBadNameShape() {
+ assertThrows(IllegalArgumentException.class,
+ () -> Layer.registerDynamic("badName", 100_005, true));
+ assertThrows(IllegalArgumentException.class,
+ () -> Layer.registerDynamic("9_LAYER", 100_006, true));
+ assertThrows(IllegalArgumentException.class,
+ () -> Layer.registerDynamic("BAD-NAME", 100_007, true));
+ assertThrows(IllegalArgumentException.class,
+ () -> Layer.registerDynamic(null, 100_008, true));
+ }
+
+ @Test
+ void unregisterDynamicRemovesRegisteredLayer() {
+ Layer.registerDynamic("DYN_REMOVE_ME", 100_009, true);
+ assertNotSame(Layer.UNDEFINED, Layer.nameOf("DYN_REMOVE_ME"));
+
+ Layer.unregisterDynamic("DYN_REMOVE_ME");
+
+ // After removal the lookup falls back to UNDEFINED via nameOf, and valueOf(int) throws.
+ assertSame(Layer.UNDEFINED, Layer.nameOf("DYN_REMOVE_ME"));
+ assertThrows(org.apache.skywalking.oap.server.core.UnexpectedException.class,
+ () -> Layer.valueOf(100_009));
+ }
+
+ @Test
+ void unregisterDynamicIsNoOpOnUnknownName() {
+ // No throw, no side effects observable through nameOf.
+ Layer.unregisterDynamic("DYN_NEVER_REGISTERED");
+ assertSame(Layer.UNDEFINED, Layer.nameOf("DYN_NEVER_REGISTERED"));
+ // Null is also tolerated.
+ Layer.unregisterDynamic(null);
+ }
+
+ @Test
+ void unregisterDynamicRefusesToRemoveBundledLayer() {
+ // MESH is a built-in registered through Layer.register — must NOT be removable
+ // through the runtime path, because its ordinal is baked into persisted
+ // ServiceTraffic primary keys. Ownership is tracked explicitly through
+ // DYNAMIC_NAMES, not inferred from the ordinal range.
+ final IllegalStateException ex = assertThrows(
+ IllegalStateException.class,
+ () -> Layer.unregisterDynamic("MESH"));
+ assert ex.getMessage().contains("MESH");
+ assert ex.getMessage().contains("non-dynamic");
+ // The built-in is still in the registry after the failed removal attempt.
+ assertSame(Layer.MESH, Layer.nameOf("MESH"));
+ }
+
+ @Test
+ void isDynamicReflectsRegistrationChannel() {
+ // Layer registered through registerDynamic is marked dynamic.
+ Layer.registerDynamic("DYN_ISDYN_PROBE", 100_020, true);
+ assertTrue(Layer.isDynamic("DYN_ISDYN_PROBE"),
+ "registerDynamic-added layer must report isDynamic=true");
+ // Built-in layer is not dynamic.
+ assertFalse(Layer.isDynamic("MESH"),
+ "built-in MESH must report isDynamic=false");
+ // Unknown name is not dynamic.
+ assertFalse(Layer.isDynamic("__NO_SUCH_LAYER__"));
+ assertFalse(Layer.isDynamic(null));
+ // After unregister the marker is cleared.
+ Layer.unregisterDynamic("DYN_ISDYN_PROBE");
+ assertFalse(Layer.isDynamic("DYN_ISDYN_PROBE"),
+ "unregisterDynamic must clear the dynamic marker");
+ }
+
+ @Test
+ void unregisterDynamicRefusesToRemoveAnOperatorLayerAtRuntimeRange() {
+ // A layer registered through Layer.register (the boot-time channel) at an ordinal
+ // that happens to fall in the runtime range MUST still be refused — the previous
+ // ordinal-only check would have wrongly accepted this case (operator yaml is
+ // recommended to use 10_000-99_999, but the registry's enforced range is
+ // anything-non-colliding). The DYNAMIC_NAMES set is the source of truth.
+ Layer.register("DYN_OPERATOR_AT_RUNTIME_TIER", 100_021, true);
+ final IllegalStateException ex = assertThrows(
+ IllegalStateException.class,
+ () -> Layer.unregisterDynamic("DYN_OPERATOR_AT_RUNTIME_TIER"));
+ assert ex.getMessage().contains("DYN_OPERATOR_AT_RUNTIME_TIER");
+ assert ex.getMessage().contains("non-dynamic");
+ // Still in the registry, still resolvable.
+ assertNotSame(Layer.UNDEFINED, Layer.nameOf("DYN_OPERATOR_AT_RUNTIME_TIER"));
+ }
+
+ @Test
+ void registerDynamicAfterUnregisterReusesOrdinal() {
+ // A runtime rule introduces a layer.
+ final Layer first = Layer.registerDynamic("DYN_REUSE_CYCLE", 100_010, true);
+ assertSame(first, Layer.valueOf(100_010));
+
+ // The declaring rule is removed; refcount drops to 0; the layer is freed.
+ Layer.unregisterDynamic("DYN_REUSE_CYCLE");
+
+ // A subsequent registration with the same (name, ordinal, normal) is permitted —
+ // the slot is free.
+ final Layer second = Layer.registerDynamic("DYN_REUSE_CYCLE", 100_010, true);
+ assertNotSame(first, second);
+ assertEquals(100_010, second.value());
+ }
+}
diff --git a/oap-server/server-starter/src/main/resources/layer-extensions.yml b/oap-server/server-starter/src/main/resources/layer-extensions.yml
index b6ade772a9..9214cb94bc 100644
--- a/oap-server/server-starter/src/main/resources/layer-extensions.yml
+++ b/oap-server/server-starter/src/main/resources/layer-extensions.yml
@@ -24,20 +24,35 @@
# every OAP node in a cluster reads and writes the same data, every node MUST agree on
# the (name, ordinal) mapping. Deploy this file identically to every node.
#
-# Recommended ordinal range:
-# 0-999 reserved by convention for the OAP distribution's built-in layers.
-# >= 1000 strongly recommended for external layers. Staying above 1000 avoids
-# colliding with future built-in layers added by the OAP distribution.
-# Conflicting ordinals or names cause OAP boot to fail with a clear error.
+# Ordinal-range convention (enforced for the runtime tier, recommended for the others):
+#
+# 0 – 9_999 built-in Layer.* constants. Currently 0..49 in use; the rest
+# of the range is reserved for future built-ins added by the
+# OAP distribution.
+# 10_000 – 99_999 boot-time external layers: this file (layer-extensions.yml),
+# LayerExtension SPI, and bundled MAL/LAL `layerDefinitions:`
+# blocks. Loaded before the registry is sealed.
+# 100_000 – 2_147_483_647
+# runtime DSL dynamic layers — introduced by MAL/LAL rules
+# pushed through the runtime-rule hot-update API. The ordinal
+# is operator-pinned (same as the other tiers — the OAP does
+# NOT auto-allocate; ordinals end up in persisted
+# ServiceTraffic primary keys and must be operator-stable
+# across restarts). Tracked through `RuntimeLayerRegistry`
+# with refcounted lifetime. NOT for use in this file.
+#
+# Conflicting ordinals or names cause OAP boot to fail with a clear error. Runtime DSL
+# rules that pin `ordinal:` below 100_000 are rejected at apply time with the
+# `layer_ordinal_out_of_range` status in the HTTP response.
#
# Name shape: must match the regex `[A-Z][A-Z0-9_]*` (upper-snake-case, leading letter).
#
# Field reference:
# name: upper-snake-case identifier. MAL/LAL rules and UI templates reference
# the layer by this name.
-# ordinal: stable integer. Once assigned and used to write data, this value is
-# permanent — changing it orphans every previously stored row tagged with
-# the old ordinal.
+# ordinal: stable integer in 10_000-99_999 for entries declared here. Once assigned
+# and used to write data, this value is permanent — changing it orphans
+# every previously stored row tagged with the old ordinal.
# normal: true = services in this layer are agent-installed
# false = services are conjectured/virtual (e.g. inferred from upstream
# calls). Defaults to true if omitted.
@@ -46,18 +61,22 @@
#
# layers:
# - name: IOT_FLEET
-# ordinal: 1000
+# ordinal: 10000
# normal: true
# - name: IOT_GATEWAY
-# ordinal: 1001
+# ordinal: 10001
# normal: true
# - name: VIRTUAL_IOT_DEVICE
-# ordinal: 1002
+# ordinal: 10002
# normal: false
#
# Other places a custom layer can be declared:
-# - At the top of a MAL or LAL rule file via a `layerDefinitions:` block —
-# recommended when a layer ships together with the rules that produce its telemetry.
-# - As an in-process extension contributed by a plugin jar on the OAP classpath.
+# - At the top of a bundled MAL or LAL rule file via a `layerDefinitions:` block —
+# loaded at OAP boot, same 10_000-99_999 ordinal tier as this file.
+# - At the top of a runtime MAL or LAL rule pushed via the hot-update API — operator
+# pins an explicit ordinal in the 100_000+ tier; refcounted across declaring rules;
+# unregistered when the last declaring rule is removed.
+# - As an in-process extension contributed by a plugin jar on the OAP classpath via
+# the LayerExtension SPI (boot-time tier).
layers: []
diff --git a/test/e2e-v2/cases/runtime-rule/mal-storage/banyandb/e2e.yaml b/test/e2e-v2/cases/runtime-rule/mal-storage/banyandb/e2e.yaml
index af1fcededf..fb8033a540 100644
--- a/test/e2e-v2/cases/runtime-rule/mal-storage/banyandb/e2e.yaml
+++ b/test/e2e-v2/cases/runtime-rule/mal-storage/banyandb/e2e.yaml
@@ -22,9 +22,10 @@
# the runtime-rule HTTP handler accepts seed rules that derive SkyWalking
# metrics from those signals; the flow script drives the full lifecycle —
# CREATE → UPDATE-FILTER → UPDATE-STRUCTURAL → DUMP → ILLEGAL × 4 →
-# SHAPE-BREAK → INACTIVATE → ACTIVATE → DELETE → DUMP — and asserts each
-# phase against /list, swctl, the BanyanDB-side measure existence, and the
-# step-label-attributed query results.
+# LAYER-REJECTION × 3 → LAYER-ROUND-TRIP → SHAPE-BREAK → INACTIVATE →
+# ACTIVATE → DELETE → DUMP — and asserts each phase against /list, swctl,
+# the BanyanDB-side measure existence, and the step-label-attributed
+# query results.
setup:
env: compose
diff --git a/test/e2e-v2/cases/runtime-rule/mal-storage/runtime-rule-flow.sh b/test/e2e-v2/cases/runtime-rule/mal-storage/runtime-rule-flow.sh
index 7f80d2b032..97831551aa 100755
--- a/test/e2e-v2/cases/runtime-rule/mal-storage/runtime-rule-flow.sh
+++ b/test/e2e-v2/cases/runtime-rule/mal-storage/runtime-rule-flow.sh
@@ -23,11 +23,15 @@
# 2. UPDATE-FILTER — POST rule v2 (body change ×10, same shape)
# 3. UPDATE-STRUCTURAL — POST rule v3 (adds 2nd metric)
# 4. DUMP (mid-flight) — GET /dump returns tar.gz with the live ruleset
-# 5. ILLEGAL-APPLY × 4 — verify rejection paths
+# 5. ILLEGAL-APPLY — verify rejection paths
# 5a. malformed YAML → 400 compile_failed
# 5b. shape flip without allowStorageChange → 409
# 5c. /delete on ACTIVE row → 409 requires_inactivate_first
# 5d. sibling rule claims the same metric name → 409 ownership conflict
+# 5e. layerDefinitions ordinal <100_000 → 400 layer_ordinal_out_of_range
+# 5f. layerDefinitions name shape invalid → 400 layer_name_invalid
+# 5g. layerDefinitions redeclares built-in MESH → 400 layer_name_conflict
+# 5h. HAPPY-PATH dynamic-LAYER round-trip via swctl `layer ls`
# 6. SHAPE-BREAK — /inactivate → /delete → POST rule v4 (INSTANCE-scope)
# 7. INACTIVATE — POST /inactivate (soft-pause)
# 8. ACTIVATE — re-POST /addOrUpdate (lossless reactivate)
@@ -504,6 +508,113 @@ list_no_row "${SIBLING_NAME}"
|| fail "5d: primary rule's contentHash moved after sibling-conflict rejection"
assert_metric_step_advanced "e2e_rr_requests" "structural" "${struct_baseline}" 180
+# Phase 5e/f/g — dynamic-LAYER rejection paths. Each fixture targets one of the
+# new applyStatus codes (layer_ordinal_out_of_range, layer_name_invalid,
+# layer_name_conflict). Same invariant as 5a-d: response JSON carries the
+# expected applyStatus, /list shows no sibling row, the primary rule's
+# contentHash is unchanged, and structural-bucket aggregation keeps advancing.
+
+log "=== Phase 5e: ILLEGAL layer ordinal below runtime floor ==="
+struct_baseline="$(latest_bucket_id_for_step "e2e_rr_requests" "structural")"
+resp="$(post_rule_expect_status \
+ "${SEED_RULES_DIR}/illegal-layer-out-of-range.yaml" "400" "" "${SIBLING_NAME}")"
+assert_apply_status "layer_ordinal_out_of_range" "${resp}"
+list_no_row "${SIBLING_NAME}"
+[[ "$(list_row ACTIVE | jq -r '.contentHash')" == "${hash_structural}" ]] \
+ || fail "5e: primary rule's contentHash moved after layer-ordinal rejection"
+assert_metric_step_advanced "e2e_rr_requests" "structural" "${struct_baseline}" 180
+
+log "=== Phase 5f: ILLEGAL layer name shape ==="
+struct_baseline="$(latest_bucket_id_for_step "e2e_rr_requests" "structural")"
+resp="$(post_rule_expect_status \
+ "${SEED_RULES_DIR}/illegal-layer-name-invalid.yaml" "400" "" "${SIBLING_NAME}")"
+assert_apply_status "layer_name_invalid" "${resp}"
+list_no_row "${SIBLING_NAME}"
+[[ "$(list_row ACTIVE | jq -r '.contentHash')" == "${hash_structural}" ]] \
+ || fail "5f: primary rule's contentHash moved after layer-name-invalid rejection"
+assert_metric_step_advanced "e2e_rr_requests" "structural" "${struct_baseline}" 180
+
+log "=== Phase 5g: ILLEGAL layer name conflict (built-in MESH redeclared) ==="
+struct_baseline="$(latest_bucket_id_for_step "e2e_rr_requests" "structural")"
+resp="$(post_rule_expect_status \
+ "${SEED_RULES_DIR}/illegal-layer-name-conflict.yaml" "400" "" "${SIBLING_NAME}")"
+assert_apply_status "layer_name_conflict" "${resp}"
+# Message must name the conflicting source so operators see what to align with.
+echo "${resp}" | jq -e '.message | test("built-in")' >/dev/null \
+ || fail "5g: response message did not label source as built-in: ${resp}"
+list_no_row "${SIBLING_NAME}"
+[[ "$(list_row ACTIVE | jq -r '.contentHash')" == "${hash_structural}" ]] \
+ || fail "5g: primary rule's contentHash moved after layer-name-conflict rejection"
+assert_metric_step_advanced "e2e_rr_requests" "structural" "${struct_baseline}" 180
+
+# Phase 5h — HAPPY-PATH dynamic-LAYER round-trip + RESTART survival.
+# POST a sibling pure-runtime rule (no bundled twin) with layerDefinitions,
+# verify swctl layer ls lists it, RESTART the OAP container, verify the
+# layer survives the restart AND remains operator-removable through
+# /inactivate + /delete. Proves the runtime-channel ownership contract
+# end-to-end (the bug RuleSetMerger.merge fix addresses).
+log "=== Phase 5h: HAPPY-PATH + RESTART dynamic-LAYER round-trip ==="
+struct_baseline="$(latest_bucket_id_for_step "e2e_rr_requests" "structural")"
+resp="$(post_rule "${SEED_RULES_DIR}/seed-rule-sibling-with-layer.yaml" "" "${SIBLING_NAME}")"
+assert_apply_status "structural_applied" "${resp}"
+list_row "ACTIVE" "${SIBLING_NAME}" >/dev/null
+sleep 2
+layers_after_create="$(swctl --display yaml \
+ --base-url=http://${OAP_HOST}:${OAP_GQL_PORT}/graphql layer ls)"
+echo "${layers_after_create}" | grep -q "E2E_RR_DYN_LAYER" \
+ || fail "5h-create: layer ls missing E2E_RR_DYN_LAYER (got: ${layers_after_create})"
+log " pre-restart: layer ls includes E2E_RR_DYN_LAYER"
+
+# Restart the OAP container. base-compose.yml pins the OAP image to
+# `skywalking/oap:latest`; filter by that. Do NOT fall back to a bare
+# `grep oap` match — on a shared dev/CI host that can hit unrelated
+# containers (e.g. another OAP, an agent test rig).
+log " restarting OAP container..."
+oap_container="$(docker ps --filter "ancestor=skywalking/oap:latest" \
+ --format '{{.Names}}' | head -1)"
+[ -n "${oap_container}" ] || fail "5h: no skywalking/oap:latest container found — refusing to guess"
+docker restart "${oap_container}" >/dev/null
+# Wait for the REST port to come back. Cap at 180s so a true hang surfaces.
+for i in $(seq 1 90); do
+ if curl -fsS "${REST_BASE}/runtime/rule/list" >/dev/null 2>&1; then
+ log " OAP back up after ${i}*2s"
+ break
+ fi
+ sleep 2
+done
+curl -fsS "${REST_BASE}/runtime/rule/list" >/dev/null \
+ || fail "5h: OAP did not come back online after restart"
+
+# Critical assertion: the runtime layer must still be visible AND retain its
+# dynamic ownership (i.e. removable through /delete below). Without the
+# RuleSetMerger override-only fix, the pure-runtime rule's layerDefinitions
+# would be replayed through Layer.register at boot and the layer would be
+# stuck for the OAP process lifetime.
+sleep 2
+layers_after_restart="$(swctl --display yaml \
+ --base-url=http://${OAP_HOST}:${OAP_GQL_PORT}/graphql layer ls)"
+echo "${layers_after_restart}" | grep -q "E2E_RR_DYN_LAYER" \
+ || fail "5h-restart: layer ls missing E2E_RR_DYN_LAYER after restart (got: ${layers_after_restart})"
+list_row "ACTIVE" "${SIBLING_NAME}" >/dev/null
+log " post-restart: layer + rule survived"
+
+# Now prove the layer is still removable through the dynamic channel.
+retry_curl_post "${REST_BASE}/runtime/rule/inactivate?catalog=${CATALOG}&name=${SIBLING_NAME}" >/dev/null \
+ || fail "5h: sibling inactivate failed"
+retry_curl_post "${REST_BASE}/runtime/rule/delete?catalog=${CATALOG}&name=${SIBLING_NAME}" >/dev/null \
+ || fail "5h: sibling delete failed"
+list_no_row "${SIBLING_NAME}"
+sleep 2
+layers_after_delete="$(swctl --display yaml \
+ --base-url=http://${OAP_HOST}:${OAP_GQL_PORT}/graphql layer ls)"
+echo "${layers_after_delete}" | grep -q "E2E_RR_DYN_LAYER" \
+ && fail "5h-delete: layer ls still includes E2E_RR_DYN_LAYER after sibling delete — runtime ownership lost across restart (RuleSetMerger merger bug?)"
+
+# Primary rule's contentHash unchanged throughout; aggregation resumed after restart.
+[[ "$(list_row ACTIVE | jq -r '.contentHash')" == "${hash_structural}" ]] \
+ || fail "5h: primary rule's contentHash moved during dynamic-layer round-trip"
+assert_metric_step_advanced "e2e_rr_requests" "structural" "${struct_baseline}" 300
+
# Phase 6 — SHAPE-BREAK via the supported route: /inactivate → /delete →
# POST a new shape under the same (catalog, name).
log "=== Phase 6: SHAPE-BREAK ==="
diff --git a/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/illegal-layer-name-conflict.yaml b/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/illegal-layer-name-conflict.yaml
new file mode 100644
index 0000000000..1272618e82
--- /dev/null
+++ b/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/illegal-layer-name-conflict.yaml
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Illegal fixture for Phase 5g: a runtime-DSL layerDefinitions entry
+# redeclaring the built-in MESH layer (ordinal 1) at a runtime ordinal
+# (200_000). The triples differ, so the REST handler must reject with HTTP
+# 400 and applyStatus `layer_name_conflict`, naming the source as
+# "built-in or boot-time extension".
+metricPrefix: e2e_rr_dyn_nc
+layerDefinitions:
+ - name: MESH
+ ordinal: 200000
+ normal: true
+expSuffix: service(['service_name'], Layer.GENERAL)
+metricsRules:
+ - name: nc_test
+ exp: e2e_rr_request_count_total.sum(['service_name'])
diff --git a/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/illegal-layer-name-invalid.yaml b/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/illegal-layer-name-invalid.yaml
new file mode 100644
index 0000000000..8f31030098
--- /dev/null
+++ b/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/illegal-layer-name-invalid.yaml
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Illegal fixture for Phase 5f: a runtime-DSL layerDefinitions entry with a
+# name that contains a hyphen — must match [A-Z][A-Z0-9_]*. The REST handler
+# must reject with HTTP 400 and applyStatus `layer_name_invalid`.
+metricPrefix: e2e_rr_dyn_inv
+layerDefinitions:
+ - name: bad-name
+ ordinal: 200000
+ normal: true
+expSuffix: service(['service_name'], Layer.GENERAL)
+metricsRules:
+ - name: inv_test
+ exp: e2e_rr_request_count_total.sum(['service_name'])
diff --git a/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/illegal-layer-out-of-range.yaml b/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/illegal-layer-out-of-range.yaml
new file mode 100644
index 0000000000..74b1ffb04f
--- /dev/null
+++ b/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/illegal-layer-out-of-range.yaml
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Illegal fixture for Phase 5e: a runtime-DSL layerDefinitions entry with
+# `ordinal: 99999` — the last slot of the boot-time external tier, one slot
+# below the 100_000 runtime floor. This is the most realistic operator
+# confusion: "external layers live in 10_000-99_999, runtime is just more
+# of the same". The runtime-rule REST handler must reject this with HTTP
+# 400 and applyStatus `layer_ordinal_out_of_range`. The metric body is
+# well-formed so the rejection lands on the layer-validation gate, not on
+# a MAL compile error. `normal:` is omitted to confirm the default (true)
+# is honored on the rejection path too.
+metricPrefix: e2e_rr_dyn_ooor
+layerDefinitions:
+ - name: E2E_RR_BAD_RANGE
+ ordinal: 99999
+expSuffix: service(['service_name'], Layer.GENERAL)
+metricsRules:
+ - name: ooor_test
+ exp: e2e_rr_request_count_total.sum(['service_name'])
diff --git a/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/seed-rule-sibling-with-layer.yaml b/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/seed-rule-sibling-with-layer.yaml
new file mode 100644
index 0000000000..58c5f6d331
--- /dev/null
+++ b/test/e2e-v2/cases/runtime-rule/mal-storage/seed-rules/seed-rule-sibling-with-layer.yaml
@@ -0,0 +1,33 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Happy-path fixture for Phase 5h: a runtime-DSL rule that introduces a NEW
+# layer through `layerDefinitions:` with an explicit `ordinal:` in the
+# runtime tier (>=100_000). `normal:` is intentionally omitted to exercise
+# the LayerDefinition default of `true` (mirrors yaml-extensions + bundled
+# MAL/LAL conventions). The metric body uses the built-in GENERAL layer so
+# this fixture only proves the layer registration, not the cross-layer
+# ingest path (which is covered by the dedicated MAL/L1 unit tests). After
+# /addOrUpdate succeeds, the flow asserts `swctl layer ls` lists
+# E2E_RR_DYN_LAYER; after /delete, the same query proves the layer is
+# unregistered.
+metricPrefix: e2e_rr_dyn_layer
+layerDefinitions:
+ - name: E2E_RR_DYN_LAYER
+ ordinal: 100050
+expSuffix: service(['service_name'], Layer.GENERAL)
+metricsRules:
+ - name: dyn_layer_carrier
+ exp: e2e_rr_request_count_total.sum(['service_name'])