You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "wu-sheng (via GitHub)" <gi...@apache.org> on 2026/08/06 23:14:11 UTC
[PR] Support runtime rule hot-update and DSL debugging for meter-analyzer-config [skywalking]
wu-sheng opened a new pull request, #13969:
URL: https://github.com/apache/skywalking/pull/13969
### Support runtime rule hot-update and DSL debugging for the `meter-analyzer-config` catalog
- [x] If this is non-trivial feature, paste the links/URLs to the design doc.
- Design doc for the mechanism this catalog now joins: [Runtime rule hot-update](https://github.com/apache/skywalking/blob/master/docs/en/concepts-and-designs/runtime-rule-hot-update.md)
- [x] Update the documentation to include this new feature.
- `docs/en/setup/backend/backend-meter.md`, `docs/en/concepts-and-designs/runtime-rule-hot-update.md`, `docs/en/setup/backend/admin-api/runtime-rule.md`, `docs/en/setup/backend/admin-api/dsl-debugging.md`, `docs/en/setup/backend/admin-api/dsl-debugging-mal.md`
- [x] Tests(including UT, IT, E2E) are added to verify the new feature.
- UT: new `MeterProcessServiceConverterRegistryTest`; `MeterProcessorTest` retargeted onto the shared loader.
- E2E: new `test/e2e-v2/cases/runtime-rule/meter/` (readiness + 5 capability phases) plus a CI matrix entry.
- [ ] If it's UI related, attach the screenshots below.
- [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #<issue number>.
- [x] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).
---
### Target
Give the `meter-analyzer-config` catalog the same debugging context and hot-update
lifecycle `otel-rules` already has. `otel-rules` behaviour is the reference and does
not change; where the meter path fell short, the meter path moved.
| capability | otel-rules | meter (before) | meter (now) |
|---|---|---|---|
| named generated classes | `Rules.parseRule` sets `name` | `getSourceName()` null → `MalExpr_<N>` | same as otel |
| DSL-debug session can bind | `MalStaticBindingHook.publish` | no publish at all | publishes |
| `Catalog` entry → `RuleKey` resolves | `OTEL_RULES` | absent | `METER_ANALYZER_CONFIG` |
| `RuleSetMerger` / `StaticRuleRegistry` | yes | bypassed (raw snakeyaml) | yes |
| hot-swappable converter | `MalConverterRegistry` | plain `List<MetricConvert>` | `MalConverterRegistry` |
### Principle
Same management, same compiler, same runtime — only the inputs differ. Rather than
teach `MeterConfig` each capability one at a time, meter rules now load through the
*identical* `Rules.loadRules` + `Rule` model otel uses, and `MeterConfig`/`MeterConfigs`
are deleted. One MAL rule model repo-wide, so this divergence cannot silently return.
**No compiler changes** (`meter-analyzer/.../compiler/**`, `.../compiler/rt/**`,
`.../dsl/**` untouched). **No MAL runtime changes.**
Out of scope, noted for a future PR: `MetricConvert.java:135` derives the class-name
`L<n>` segment from the rules-list *index*, not the YAML line, so production emits
`vm_L0_cpu_total_percentage` while the test harness emits `vm_L37_…`. Real, but it
affects every MAL catalog and is not this change's agenda.
### Scope
- **Catalog registration** — `Catalog`, `MalRuleEngine` (`CATALOGS` +
`resolveConverterRegistry` → `agent-analyzer`), `DbOverrideRuntimeRuleResolver`,
`MALHolderRegistry`, `MALDebugRecorderFactory`, javadoc/proto catalog lists.
- **agent-analyzer** — `MeterProcessService` implements `MalConverterRegistry`
(volatile copy-on-write map, same shape as the otel receiver) and publishes
`MalStaticBindingHook`; registered on `AnalyzerModule` under both contracts; rule
loading moved `prepare()` → `start()` so the override resolver sees a live storage
module; `MeterProcessor` re-reads `converts()` per batch, so a hot-add lands on the
next batch with no restart.
- **Tests** — `MeterProcessorTest` retargeted (3 lines, assertions unchanged); new
`MeterProcessServiceConverterRegistryTest` mirroring the otel registry contract test.
- **E2E** — `test/e2e-v2/cases/runtime-rule/meter/` + one CI matrix entry. Six phases,
each guarding one thing this PR changes:
0. admin API reachable;
1. **bundled visibility** — shipped `batch-meter.yaml` appears in `runtime-rule list`
(proves `StaticRuleRegistry` participation; fails on master);
2. **hot add** — a twin-less runtime rule becomes queryable with no restart;
3. **edit** — re-applying the same `(catalog, name)` with an extra metric registers the
new metric *and* keeps the already-registered one flowing (converter **replacement**);
4. **DSL debug** — a session binds a *bundled* meter rule and captures records carrying
verbatim DSL, real samples and a terminal `meterEmit` (proves the MAL catalog
predicates plus `MeterProcessService`'s boot-time `MalStaticBindingHook` publish);
5. **inactivate** — row goes explicitly `INACTIVE`, its metrics stop producing new
buckets, bundled keeps producing (converter **removal**).
Freshness: every "flowing / stopped" assertion diffs the set of time-bucket ids carrying
a non-null value against a baseline taken at the transition, so a metric cannot pass on
stale pre-transition buckets and a stopped metric cannot be masked by history. Bucket-id
deltas are used instead of absolute `--start/--end` so the assertions are immune to
host-vs-container clock/timezone skew.
Failure vs. absence: the bucket-reading helper returns a non-zero status for transport
errors, non-JSON bodies and malformed envelopes, which is deliberately distinct from a
successful query that found no data. The negative ("has it stopped") assertion counts
only successful observations and refuses to conclude anything below a floor of 3, so a
broken query path fails the test instead of silently satisfying it. No existing e2e case
was modified.
- **Docs** — 5 docs + changelog.
- **Drive-by fix** — `library-pprof-parser` was the only module in the repo with
`<checkstyle.skip>true</checkstyle.skip>`, hiding 2 real violations. Removed the
exemption and fixed the violations; full-repo checkstyle is clean.
### Behaviour change
An entry in `meterAnalyzerActiveFiles` / `SW_METER_ANALYZER_ACTIVE_FILES` with no
matching rule file now **fails OAP startup** instead of being silently ignored. The
list is operator-supplied, so the files must exist; this also matches how `otel-rules`
has always behaved. Documented in `backend-meter.md` and the changelog.
### Verification
Passing locally: compile, full-repo `checkstyle:check`, `license-eye` (0 invalid /
4969 files), `MeterProcessorTest` + `MeterProcessServiceConverterRegistryTest` (4/4),
`docker compose config` (mounts and the 3-level relative jar path resolve), `bash -n`,
and the bucket-delta helper unit-tested offline across 5 cases — including the critical
"stale buckets only must NOT count as fresh".
**Not run locally: the e2e case itself** — needs a built OAP image and a full compose
cycle. Static validation only; watch it in CI.
### Review
A multi-agent adversarial review over the whole change set raised 15 candidate findings;
12 were refuted on inspection, 3 confirmed and fixed here:
- **Missing `StorageModule` dependency (real bug, introduced by this PR).** Meter rule
loading moved to `start()` precisely because `RuleSetMerger` consults the runtime-rule
DB override resolver — but `AnalyzerModuleProvider.requiredModules()` never declared
`StorageModule`. A boot-order simulation over all 40 default-enabled modules put
`agent-analyzer` at sequence 6 and `storage` at 13, so `RuntimeRuleManagementDAO.getAll()`
would throw "management schema not registered yet", the resolver would log a WARN and
return empty, and an operator's meter-rule override / inactivate would be ignored until
the reconciler's next tick. Every other `Rules.loadRules` caller (otel, telegraf, envoy,
log-analyzer) already declares it. Fixed, with the same rationale comment; no cycle
(storage requires only `CoreModule`).
- **E2E raced the async apply.** Phase 2 read rule status once immediately after
`addOrUpdate`; the apply is async (durable commit returns before the schema fence rolls
out). Now polls to `ACTIVE`, as `dsl-debugging/mal` does — and likewise for the phase-3
edit and the phase-5 inactivate.
- **Budget vs. timeout.** Trimmed internal budgets so the pathological worst case is
~27.5m inside a 40m case timeout. This matters beyond tidiness: a harness timeout
*cancels* the job, and `upload-artifact: if failure()` does not run on cancellation, so
a timeout kill yields no logs. Internal budgets must always trip first.
### Follow-up (separate repo)
`skywalking-cli` help text lists only 4 catalogs. The `Catalogs` vars are declared but
never referenced, so `--catalog meter-analyzer-config` already works — this is
cosmetic: `pkg/admin/runtimerule/runtimerule.go:35`,
`pkg/admin/dsldebug/dsldebug.go:40`,
`internal/commands/admin/runtimerule/runtimerule.go:39,54`,
`internal/commands/admin/dsldebug/dsldebug.go:89`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support runtime rule hot-update and DSL debugging for meter-analyzer-config [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng merged PR #13969:
URL: https://github.com/apache/skywalking/pull/13969
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support runtime rule hot-update and DSL debugging for meter-analyzer-config [skywalking]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13969:
URL: https://github.com/apache/skywalking/pull/13969#discussion_r3732444619
##########
test/e2e-v2/cases/runtime-rule/meter/meter-runtime-rule-flow.sh:
##########
@@ -0,0 +1,416 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Runtime-rule hot-update over the `meter-analyzer-config` catalog.
+#
+# A readiness phase followed by five capability phases, each guarding something this
+# catalog did not have before meter rules started loading through meter-analyzer's
+# Rules/Rule pipeline:
+#
+# 0. Admin API reachable (readiness only, asserts no capability).
+# 1. BUNDLED VISIBILITY — the shipped batch-meter.yaml appears in
+# `runtime-rule list`. Only true once meter rules go through
+# RuleSetMerger and are recorded in StaticRuleRegistry. Previously the
+# meter loader read YAML directly, so the catalog was invisible here.
+# 2. HOT ADD — a pure-runtime meter rule (no on-disk twin) becomes a live,
+# queryable metric with no OAP restart. Proves the applied MetricConvert
+# reached MeterProcessService's MalConverterRegistry and that
+# MeterProcessor picks it up on the next batch.
+# 3. EDIT — re-applying the same (catalog, name) with an extra metric
+# registers the new metric AND keeps the already-registered one flowing.
+# Proves converter REPLACEMENT, not just first insertion.
+# 4. DSL DEBUG — a debug session binds a BUNDLED meter rule and captures
+# records carrying the verbatim DSL, real samples, and a terminal
+# meterEmit. Proves the MAL catalog predicates accept the meter catalog
+# and that MeterProcessService publishes its holders at boot.
+# 5. INACTIVATE — the runtime rule is soft-paused: the row goes INACTIVE and
+# its metrics stop producing new buckets, while the bundled rule keeps
+# producing. Proves converter REMOVAL.
+#
+# FRESHNESS: every "is it flowing / has it stopped" assertion compares the SET
+# OF TIME-BUCKET IDS carrying a non-null value against a baseline captured at
+# the transition. A metric that merely retains old buckets from before the
+# transition therefore does NOT satisfy "still flowing", and a stopped metric
+# cannot be masked by history. Bucket ids are compared instead of absolute
+# --start/--end timestamps so the assertions are immune to host-vs-container
+# clock and timezone skew.
+#
+set -euo pipefail
+
+OAP_HOST="${OAP_HOST:-127.0.0.1}"
+OAP_REST_PORT="${OAP_REST_PORT:-17128}"
+OAP_GQL_PORT="${OAP_GQL_PORT:-12800}"
+SENDER_HOST="${SENDER_HOST:-127.0.0.1}"
+SENDER_PORT="${SENDER_PORT:-9093}"
+SEED_DIR="${SEED_DIR:-./seed-rules}"
+
+CATALOG="meter-analyzer-config"
+BUNDLED_NAME="batch-meter"
+RUNTIME_NAME="e2e_rr_meter"
+
+# Metric emitted by the bundled rule (metricPrefix `batch` + rule `test`).
+BUNDLED_METRIC="batch_test"
+# Metric emitted by the runtime rule (metricPrefix `e2e_rr_meter` + rule `batch`).
+RUNTIME_METRIC="e2e_rr_meter_batch"
+# Second metric the v2 EDIT of that same rule adds.
+RUNTIME_METRIC_V2="e2e_rr_meter_batch_scaled"
+
+# The mock sender always reports under this identity.
+SVC="test-service"
+INST="test-instance"
+
+REST_BASE="http://${OAP_HOST}:${OAP_REST_PORT}"
+GQL_BASE="http://${OAP_HOST}:${OAP_GQL_PORT}/graphql"
+
+# First-data budget: needs a minute-bucket boundary plus a persistence flush on
+# a cold pipeline.
+METRIC_BUDGET_S="${METRIC_BUDGET_S:-240}"
+# Budget for a NEW bucket once the pipeline is already warm.
+FRESH_BUDGET_S="${FRESH_BUDGET_S:-150}"
+# Grace after /inactivate for in-flight samples and the open minute bucket to
+# land, before the "has it stopped" baseline is taken.
+STOP_GRACE_S="${STOP_GRACE_S:-90}"
+# How long to keep pushing after that baseline while proving nothing new lands.
+STOP_SETTLE_S="${STOP_SETTLE_S:-120}"
+# Minimum number of SUCCESSFUL observations before "the metric stopped" may be
+# concluded. Guards the negative assertion against a vacuous pass when the query
+# path itself is broken for the whole settle window.
+MIN_STOPPED_OBSERVATIONS="${MIN_STOPPED_OBSERVATIONS:-3}"
+# Budget for an /addOrUpdate to reach ACTIVE. The apply is async — the REST call returns
+# after the durable commit, while the schema fence rolls out in the background (its own
+# default budget is 180s), so this must not be a single read.
+APPLY_LAND_S="${APPLY_LAND_S:-120}"
+# Budget for the debug session to capture its first record.
+DEBUG_BUDGET_S="${DEBUG_BUDGET_S:-150}"
+DBG_CLIENT_ID="${DBG_CLIENT_ID:-e2e-rr-meter-dbg-1}"
+
+log() { echo "[meter-runtime-rule-flow] $*" >&2; }
+fail() { echo "[meter-runtime-rule-flow] FAIL: $*" >&2; exit 1; }
+
+# Same convention as the otel-catalog flow: every runtime-rule REST call goes
+# through swctl's admin command tree. swctl passes --catalog through verbatim,
+# so `meter-analyzer-config` needs no CLI change.
+admin() { swctl --display json --admin-url="${REST_BASE}" admin "$@"; }
+
+# Push one batch of native MeterData (raw meter `batch_test`).
+push_meter() {
+ curl -s -XPOST "http://${SENDER_HOST}:${SENDER_PORT}/sendBatchMetrics" >/dev/null \
+ || fail "mock sender did not accept /sendBatchMetrics"
+}
+
+# Newline-separated, sorted set of time-bucket ids that currently carry a
+# non-null value for a metric.
+#
+# Return code is load-bearing, and MUST stay that way: a transport error, a
+# non-JSON body, or a malformed envelope returns non-zero, which is NOT the same
+# as "queried fine, metric has no data yet" (empty stdout, return 0). The
+# negative assertion below relies on that distinction — if a broken query were
+# silently reported as an empty bucket set, "no new buckets" would be
+# indistinguishable from "never managed to look", and the stopped-check would
+# pass vacuously.
+metric_buckets() {
+ local metric="$1"
+ local out
+ out="$(swctl --display json --base-url="${GQL_BASE}" metrics exec \
+ --expression="${metric}" \
+ --service-name="${SVC}" --instance-name="${INST}" 2>/dev/null)" \
+ || return 1
+ # A well-formed response is a JSON object carrying `results`. Anything else
+ # (empty body, HTML error page, truncated JSON) is a query failure, not an
+ # empty result set. `results: null` is accepted — that is the legitimate
+ # "metric not present yet" shape.
+ printf '%s' "${out}" | jq -e 'type == "object" and has("results")' >/dev/null 2>&1 \
+ || return 1
+ printf '%s' "${out}" \
+ | jq -r '[.results[]?.values[]? | select(.value != null) | .id] | sort | .[]' \
+ || return 1
+}
+
+# Ids present now that were absent from the supplied baseline. Propagates a
+# query failure as a non-zero return so callers can tell it apart from "no new
+# buckets".
+new_buckets_since() {
+ local metric="$1"
+ local baseline="$2"
+ local current
+ current="$(metric_buckets "${metric}")" || return 1
+ comm -13 <(printf '%s\n' "${baseline}" | sort -u) \
+ <(printf '%s\n' "${current}" | sort -u)
+}
+
+# Block until the metric produces ANY non-null bucket. Used only for the very
+# first data point of a cold pipeline.
+wait_metric() {
+ local metric="$1"
+ local deadline=$(( $(date +%s) + METRIC_BUDGET_S ))
+ local buckets
+ local failures=0
+ while (( $(date +%s) < deadline )); do
+ push_meter
+ sleep 10
+ if ! buckets="$(metric_buckets "${metric}")"; then
+ failures=$(( failures + 1 ))
+ continue
+ fi
+ if [[ -n "${buckets}" ]]; then
+ log " ${metric} has data"
+ return 0
+ fi
+ done
+ fail "${metric} never produced a value within ${METRIC_BUDGET_S}s (${failures} query failure(s))"
+}
+
+# Block until the metric produces a bucket NEWER than the supplied baseline.
+# This is the assertion that actually proves the converter is live right now.
+wait_fresh_metric() {
+ local metric="$1"
+ local baseline="$2"
+ local deadline=$(( $(date +%s) + FRESH_BUDGET_S ))
+ local fresh
+ local failures=0
+ while (( $(date +%s) < deadline )); do
+ push_meter
+ sleep 10
+ if ! fresh="$(new_buckets_since "${metric}" "${baseline}")"; then
+ failures=$(( failures + 1 ))
+ continue
+ fi
+ if [[ -n "${fresh}" ]]; then
+ log " ${metric} produced fresh bucket(s): $(echo "${fresh}" | tr '\n' ' ')"
+ return 0
+ fi
+ done
+ fail "${metric} produced no NEW bucket within ${FRESH_BUDGET_S}s (${failures} query failure(s)) — the converter is not live"
+}
+
+# Assert the metric produces NO bucket newer than the baseline for the whole
+# settle window, while samples keep being pushed.
+#
+# This is a NEGATIVE assertion, so "we never saw anything" must NOT count as
+# success. Every loop iteration that fails to query is recorded, and the check
+# only concludes "stopped" if at least MIN_STOPPED_OBSERVATIONS iterations
+# actually observed the metric successfully.
+assert_metric_stopped() {
+ local metric="$1"
+ local baseline="$2"
+ local deadline=$(( $(date +%s) + STOP_SETTLE_S ))
+ local observations=0
+ local failures=0
+ local fresh
+ while (( $(date +%s) < deadline )); do
+ push_meter
+ sleep 10
+ if ! fresh="$(new_buckets_since "${metric}" "${baseline}")"; then
+ failures=$(( failures + 1 ))
+ log " WARN: query for ${metric} failed (${failures} so far) — not counted as an observation"
+ continue
+ fi
+ observations=$(( observations + 1 ))
+ if [[ -n "${fresh}" ]]; then
+ fail "${metric} produced a NEW bucket after inactivate: $(echo "${fresh}" | tr '\n' ' ')"
+ fi
+ done
+ (( observations >= MIN_STOPPED_OBSERVATIONS )) \
+ || fail "cannot conclude ${metric} stopped: only ${observations} successful observation(s) in ${STOP_SETTLE_S}s (${failures} query failure(s)); need >= ${MIN_STOPPED_OBSERVATIONS}"
+ log " ${metric} produced no new buckets across ${observations} successful observation(s) — converter removed"
+}
+
+# Status string the /list row carries for a (catalog, name), or empty.
+rule_status() {
+ admin runtime-rule list 2>/dev/null \
+ | jq -r --arg c "${CATALOG}" --arg n "$1" \
+ '.rules[] | select(.catalog == $c and .name == $n) | .status' | head -1
+}
+
+# ---- phase 0: admin API reachable -----------------------------------------
+log "phase 0: waiting for the runtime-rule admin API"
+deadline=$(( $(date +%s) + 180 ))
+until admin runtime-rule list >/dev/null 2>&1; do
+ (( $(date +%s) < deadline )) || fail "runtime-rule admin API never became reachable"
+ sleep 3
+done
+log " admin API up"
+
+# ---- phase 1: bundled meter rule is visible -------------------------------
+log "phase 1: bundled ${CATALOG}/${BUNDLED_NAME} must be visible to runtime-rule"
+admin runtime-rule list \
+ | jq -e --arg c "${CATALOG}" --arg n "${BUNDLED_NAME}" \
+ '.rules[] | select(.catalog==$c and .name==$n)' >/dev/null \
+ || fail "bundled ${CATALOG}/${BUNDLED_NAME} is NOT visible in runtime-rule list — meter rules are not reaching StaticRuleRegistry"
+log " bundled rule visible"
+
+log " and the bundled metric ${BUNDLED_METRIC} produces data"
+wait_metric "${BUNDLED_METRIC}"
+
+# ---- phase 2: hot-add a pure-runtime meter rule ---------------------------
+log "phase 2: hot-add ${CATALOG}/${RUNTIME_NAME} (no on-disk twin)"
+admin runtime-rule add --catalog "${CATALOG}" --name "${RUNTIME_NAME}" \
+ -f "${SEED_DIR}/meter-v1.yaml" \
+ || fail "addOrUpdate of ${RUNTIME_NAME} returned non-2xx"
+
+# The apply is asynchronous — addOrUpdate returns once the row is persisted, while the
+# schema fence and peer roll-out complete in the background. Poll rather than reading the
+# status once, exactly as cases/dsl-debugging/mal/dsl-debug-flow.sh does.
+wait_rule_active() {
+ local name="$1"
+ local deadline=$(( $(date +%s) + APPLY_LAND_S ))
+ local status=""
+ while (( $(date +%s) < deadline )); do
+ status="$(rule_status "${name}" || true)"
+ [[ "${status}" == "ACTIVE" ]] && { log " ${CATALOG}/${name} is ACTIVE"; return 0; }
Review Comment:
`ACTIVE` is the persisted rule-row state, not the asynchronous apply state. On the phase-3 edit the existing row is already `ACTIVE`, so this returns immediately without waiting for the new structural apply; it also cannot surface `FAILED`/`DEGRADED`. Capture the `applyId` from each `add` response and poll `/runtime/rule/status` to a terminal phase, as the existing `mal-storage/runtime-rule-flow.sh` helper does.
##########
test/e2e-v2/cases/runtime-rule/meter/meter-runtime-rule-flow.sh:
##########
@@ -0,0 +1,416 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Runtime-rule hot-update over the `meter-analyzer-config` catalog.
+#
+# A readiness phase followed by five capability phases, each guarding something this
+# catalog did not have before meter rules started loading through meter-analyzer's
+# Rules/Rule pipeline:
+#
+# 0. Admin API reachable (readiness only, asserts no capability).
+# 1. BUNDLED VISIBILITY — the shipped batch-meter.yaml appears in
+# `runtime-rule list`. Only true once meter rules go through
+# RuleSetMerger and are recorded in StaticRuleRegistry. Previously the
+# meter loader read YAML directly, so the catalog was invisible here.
+# 2. HOT ADD — a pure-runtime meter rule (no on-disk twin) becomes a live,
+# queryable metric with no OAP restart. Proves the applied MetricConvert
+# reached MeterProcessService's MalConverterRegistry and that
+# MeterProcessor picks it up on the next batch.
+# 3. EDIT — re-applying the same (catalog, name) with an extra metric
+# registers the new metric AND keeps the already-registered one flowing.
+# Proves converter REPLACEMENT, not just first insertion.
+# 4. DSL DEBUG — a debug session binds a BUNDLED meter rule and captures
+# records carrying the verbatim DSL, real samples, and a terminal
+# meterEmit. Proves the MAL catalog predicates accept the meter catalog
+# and that MeterProcessService publishes its holders at boot.
+# 5. INACTIVATE — the runtime rule is soft-paused: the row goes INACTIVE and
+# its metrics stop producing new buckets, while the bundled rule keeps
+# producing. Proves converter REMOVAL.
+#
+# FRESHNESS: every "is it flowing / has it stopped" assertion compares the SET
+# OF TIME-BUCKET IDS carrying a non-null value against a baseline captured at
+# the transition. A metric that merely retains old buckets from before the
+# transition therefore does NOT satisfy "still flowing", and a stopped metric
+# cannot be masked by history. Bucket ids are compared instead of absolute
+# --start/--end timestamps so the assertions are immune to host-vs-container
+# clock and timezone skew.
+#
+set -euo pipefail
+
+OAP_HOST="${OAP_HOST:-127.0.0.1}"
+OAP_REST_PORT="${OAP_REST_PORT:-17128}"
+OAP_GQL_PORT="${OAP_GQL_PORT:-12800}"
+SENDER_HOST="${SENDER_HOST:-127.0.0.1}"
+SENDER_PORT="${SENDER_PORT:-9093}"
+SEED_DIR="${SEED_DIR:-./seed-rules}"
+
+CATALOG="meter-analyzer-config"
+BUNDLED_NAME="batch-meter"
+RUNTIME_NAME="e2e_rr_meter"
+
+# Metric emitted by the bundled rule (metricPrefix `batch` + rule `test`).
+BUNDLED_METRIC="batch_test"
+# Metric emitted by the runtime rule (metricPrefix `e2e_rr_meter` + rule `batch`).
+RUNTIME_METRIC="e2e_rr_meter_batch"
+# Second metric the v2 EDIT of that same rule adds.
+RUNTIME_METRIC_V2="e2e_rr_meter_batch_scaled"
+
+# The mock sender always reports under this identity.
+SVC="test-service"
+INST="test-instance"
+
+REST_BASE="http://${OAP_HOST}:${OAP_REST_PORT}"
+GQL_BASE="http://${OAP_HOST}:${OAP_GQL_PORT}/graphql"
+
+# First-data budget: needs a minute-bucket boundary plus a persistence flush on
+# a cold pipeline.
+METRIC_BUDGET_S="${METRIC_BUDGET_S:-240}"
+# Budget for a NEW bucket once the pipeline is already warm.
+FRESH_BUDGET_S="${FRESH_BUDGET_S:-150}"
+# Grace after /inactivate for in-flight samples and the open minute bucket to
+# land, before the "has it stopped" baseline is taken.
+STOP_GRACE_S="${STOP_GRACE_S:-90}"
+# How long to keep pushing after that baseline while proving nothing new lands.
+STOP_SETTLE_S="${STOP_SETTLE_S:-120}"
+# Minimum number of SUCCESSFUL observations before "the metric stopped" may be
+# concluded. Guards the negative assertion against a vacuous pass when the query
+# path itself is broken for the whole settle window.
+MIN_STOPPED_OBSERVATIONS="${MIN_STOPPED_OBSERVATIONS:-3}"
+# Budget for an /addOrUpdate to reach ACTIVE. The apply is async — the REST call returns
+# after the durable commit, while the schema fence rolls out in the background (its own
+# default budget is 180s), so this must not be a single read.
+APPLY_LAND_S="${APPLY_LAND_S:-120}"
+# Budget for the debug session to capture its first record.
+DEBUG_BUDGET_S="${DEBUG_BUDGET_S:-150}"
+DBG_CLIENT_ID="${DBG_CLIENT_ID:-e2e-rr-meter-dbg-1}"
+
+log() { echo "[meter-runtime-rule-flow] $*" >&2; }
+fail() { echo "[meter-runtime-rule-flow] FAIL: $*" >&2; exit 1; }
+
+# Same convention as the otel-catalog flow: every runtime-rule REST call goes
+# through swctl's admin command tree. swctl passes --catalog through verbatim,
+# so `meter-analyzer-config` needs no CLI change.
+admin() { swctl --display json --admin-url="${REST_BASE}" admin "$@"; }
+
+# Push one batch of native MeterData (raw meter `batch_test`).
+push_meter() {
+ curl -s -XPOST "http://${SENDER_HOST}:${SENDER_PORT}/sendBatchMetrics" >/dev/null \
+ || fail "mock sender did not accept /sendBatchMetrics"
+}
+
+# Newline-separated, sorted set of time-bucket ids that currently carry a
+# non-null value for a metric.
+#
+# Return code is load-bearing, and MUST stay that way: a transport error, a
+# non-JSON body, or a malformed envelope returns non-zero, which is NOT the same
+# as "queried fine, metric has no data yet" (empty stdout, return 0). The
+# negative assertion below relies on that distinction — if a broken query were
+# silently reported as an empty bucket set, "no new buckets" would be
+# indistinguishable from "never managed to look", and the stopped-check would
+# pass vacuously.
+metric_buckets() {
+ local metric="$1"
+ local out
+ out="$(swctl --display json --base-url="${GQL_BASE}" metrics exec \
+ --expression="${metric}" \
+ --service-name="${SVC}" --instance-name="${INST}" 2>/dev/null)" \
+ || return 1
+ # A well-formed response is a JSON object carrying `results`. Anything else
+ # (empty body, HTML error page, truncated JSON) is a query failure, not an
+ # empty result set. `results: null` is accepted — that is the legitimate
+ # "metric not present yet" shape.
+ printf '%s' "${out}" | jq -e 'type == "object" and has("results")' >/dev/null 2>&1 \
+ || return 1
+ printf '%s' "${out}" \
+ | jq -r '[.results[]?.values[]? | select(.value != null) | .id] | sort | .[]' \
+ || return 1
+}
+
+# Ids present now that were absent from the supplied baseline. Propagates a
+# query failure as a non-zero return so callers can tell it apart from "no new
+# buckets".
+new_buckets_since() {
+ local metric="$1"
+ local baseline="$2"
+ local current
+ current="$(metric_buckets "${metric}")" || return 1
+ comm -13 <(printf '%s\n' "${baseline}" | sort -u) \
+ <(printf '%s\n' "${current}" | sort -u)
+}
+
+# Block until the metric produces ANY non-null bucket. Used only for the very
+# first data point of a cold pipeline.
+wait_metric() {
+ local metric="$1"
+ local deadline=$(( $(date +%s) + METRIC_BUDGET_S ))
+ local buckets
+ local failures=0
+ while (( $(date +%s) < deadline )); do
+ push_meter
+ sleep 10
+ if ! buckets="$(metric_buckets "${metric}")"; then
+ failures=$(( failures + 1 ))
+ continue
+ fi
+ if [[ -n "${buckets}" ]]; then
+ log " ${metric} has data"
+ return 0
+ fi
+ done
+ fail "${metric} never produced a value within ${METRIC_BUDGET_S}s (${failures} query failure(s))"
+}
+
+# Block until the metric produces a bucket NEWER than the supplied baseline.
+# This is the assertion that actually proves the converter is live right now.
+wait_fresh_metric() {
+ local metric="$1"
+ local baseline="$2"
+ local deadline=$(( $(date +%s) + FRESH_BUDGET_S ))
+ local fresh
+ local failures=0
+ while (( $(date +%s) < deadline )); do
+ push_meter
+ sleep 10
+ if ! fresh="$(new_buckets_since "${metric}" "${baseline}")"; then
+ failures=$(( failures + 1 ))
+ continue
+ fi
+ if [[ -n "${fresh}" ]]; then
+ log " ${metric} produced fresh bucket(s): $(echo "${fresh}" | tr '\n' ' ')"
+ return 0
+ fi
+ done
+ fail "${metric} produced no NEW bucket within ${FRESH_BUDGET_S}s (${failures} query failure(s)) — the converter is not live"
+}
+
+# Assert the metric produces NO bucket newer than the baseline for the whole
+# settle window, while samples keep being pushed.
+#
+# This is a NEGATIVE assertion, so "we never saw anything" must NOT count as
+# success. Every loop iteration that fails to query is recorded, and the check
+# only concludes "stopped" if at least MIN_STOPPED_OBSERVATIONS iterations
+# actually observed the metric successfully.
+assert_metric_stopped() {
+ local metric="$1"
+ local baseline="$2"
+ local deadline=$(( $(date +%s) + STOP_SETTLE_S ))
+ local observations=0
+ local failures=0
+ local fresh
+ while (( $(date +%s) < deadline )); do
+ push_meter
+ sleep 10
+ if ! fresh="$(new_buckets_since "${metric}" "${baseline}")"; then
+ failures=$(( failures + 1 ))
+ log " WARN: query for ${metric} failed (${failures} so far) — not counted as an observation"
+ continue
+ fi
+ observations=$(( observations + 1 ))
+ if [[ -n "${fresh}" ]]; then
+ fail "${metric} produced a NEW bucket after inactivate: $(echo "${fresh}" | tr '\n' ' ')"
+ fi
+ done
+ (( observations >= MIN_STOPPED_OBSERVATIONS )) \
+ || fail "cannot conclude ${metric} stopped: only ${observations} successful observation(s) in ${STOP_SETTLE_S}s (${failures} query failure(s)); need >= ${MIN_STOPPED_OBSERVATIONS}"
+ log " ${metric} produced no new buckets across ${observations} successful observation(s) — converter removed"
+}
+
+# Status string the /list row carries for a (catalog, name), or empty.
+rule_status() {
+ admin runtime-rule list 2>/dev/null \
+ | jq -r --arg c "${CATALOG}" --arg n "$1" \
+ '.rules[] | select(.catalog == $c and .name == $n) | .status' | head -1
+}
+
+# ---- phase 0: admin API reachable -----------------------------------------
+log "phase 0: waiting for the runtime-rule admin API"
+deadline=$(( $(date +%s) + 180 ))
+until admin runtime-rule list >/dev/null 2>&1; do
+ (( $(date +%s) < deadline )) || fail "runtime-rule admin API never became reachable"
+ sleep 3
+done
+log " admin API up"
+
+# ---- phase 1: bundled meter rule is visible -------------------------------
+log "phase 1: bundled ${CATALOG}/${BUNDLED_NAME} must be visible to runtime-rule"
+admin runtime-rule list \
+ | jq -e --arg c "${CATALOG}" --arg n "${BUNDLED_NAME}" \
+ '.rules[] | select(.catalog==$c and .name==$n)' >/dev/null \
+ || fail "bundled ${CATALOG}/${BUNDLED_NAME} is NOT visible in runtime-rule list — meter rules are not reaching StaticRuleRegistry"
+log " bundled rule visible"
+
+log " and the bundled metric ${BUNDLED_METRIC} produces data"
+wait_metric "${BUNDLED_METRIC}"
+
+# ---- phase 2: hot-add a pure-runtime meter rule ---------------------------
+log "phase 2: hot-add ${CATALOG}/${RUNTIME_NAME} (no on-disk twin)"
+admin runtime-rule add --catalog "${CATALOG}" --name "${RUNTIME_NAME}" \
+ -f "${SEED_DIR}/meter-v1.yaml" \
+ || fail "addOrUpdate of ${RUNTIME_NAME} returned non-2xx"
+
+# The apply is asynchronous — addOrUpdate returns once the row is persisted, while the
+# schema fence and peer roll-out complete in the background. Poll rather than reading the
+# status once, exactly as cases/dsl-debugging/mal/dsl-debug-flow.sh does.
+wait_rule_active() {
+ local name="$1"
+ local deadline=$(( $(date +%s) + APPLY_LAND_S ))
+ local status=""
+ while (( $(date +%s) < deadline )); do
+ status="$(rule_status "${name}" || true)"
+ [[ "${status}" == "ACTIVE" ]] && { log " ${CATALOG}/${name} is ACTIVE"; return 0; }
+ sleep 3
+ done
+ fail "${CATALOG}/${name} did not reach ACTIVE within ${APPLY_LAND_S}s (last saw '${status}')"
+}
+
+wait_rule_active "${RUNTIME_NAME}"
+
+log " applied; ${RUNTIME_METRIC} must become queryable with no restart"
+wait_metric "${RUNTIME_METRIC}"
+
+# ---- phase 3: EDIT the runtime rule in place ------------------------------
+# Baselines are captured BEFORE the edit, so the post-edit assertions cannot be
+# satisfied by buckets the v1 converter already wrote.
+log "phase 3: edit ${CATALOG}/${RUNTIME_NAME} in place (v1 -> v2, adds a metric)"
+pre_edit_v1="$(metric_buckets "${RUNTIME_METRIC}")" \
+ || fail "could not read a baseline for ${RUNTIME_METRIC} before the edit"
+# v2's metric does not exist yet, so an empty set is the expected answer here.
+# Tolerate a query failure specifically because "unknown metric" may surface as
+# an error rather than an empty envelope; an empty baseline only ever makes the
+# follow-up fresh-bucket assertion easier to satisfy, never the stopped-check.
+pre_edit_v2="$(metric_buckets "${RUNTIME_METRIC_V2}" || true)"
+
+admin runtime-rule add --catalog "${CATALOG}" --name "${RUNTIME_NAME}" \
+ -f "${SEED_DIR}/meter-v2.yaml" \
+ || fail "addOrUpdate (edit) of ${RUNTIME_NAME} returned non-2xx"
+
+# Declaring an extra metric makes this a STRUCTURAL apply, so the fence/roll-out window is
+# at its widest here. Settle on ACTIVE first, so a stalled apply is reported as such rather
+# than surfacing later as a confusing "converter is not live".
+wait_rule_active "${RUNTIME_NAME}"
+
+log " the newly declared ${RUNTIME_METRIC_V2} must produce a fresh bucket"
+wait_fresh_metric "${RUNTIME_METRIC_V2}" "${pre_edit_v2}"
+
+log " and the metric v1 already registered must still produce fresh buckets"
+wait_fresh_metric "${RUNTIME_METRIC}" "${pre_edit_v1}"
Review Comment:
This assertion can pass on buckets emitted by the old v1 converter while the structural edit is still fencing/rolling out, because its baseline was captured before the edit and the v2 wait may run for several minutes. It therefore does not prove the replacement converter kept v1's metric. After waiting for the edit's `applyId` to reach a successful terminal phase, capture a new v1 baseline and then require a bucket newer than that baseline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org