[PR] Runtime rule hot-update for MAL and LAL [skywalking]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng opened a new pull request, #13851:
URL: https://github.com/apache/skywalking/pull/13851

   ### Runtime rule hot-update for MAL and LAL
   
   Operators can now hot-update OTEL MAL, log MAL, telegraf MAL, and LAL rule files
   without restarting OAP. A new admin REST endpoint on port 17128 (off by default)
   persists rule changes to management storage; every node in the cluster converges
   on the new content within ~30 s. Engine state, cluster Suspend/Resume, BanyanDB
   schema-watch fence, and per-backend dropTable lifecycle make the apply
   end-to-end safe across structural cutovers.
   
   - [ ] If this is non-trivial feature, paste the links/URLs to the design doc.
     - Concept doc: [docs/en/concepts-and-designs/runtime-rule-hot-update.md](docs/en/concepts-and-designs/runtime-rule-hot-update.md)
   - [x] Update the documentation to include this new feature.
     - REST API: [docs/en/setup/backend/backend-runtime-rule-api.md](docs/en/setup/backend/backend-runtime-rule-api.md)
     - Architecture: [docs/en/concepts-and-designs/runtime-rule-hot-update.md](docs/en/concepts-and-designs/runtime-rule-hot-update.md)
     - Security notice: [docs/en/security/README.md](docs/en/security/README.md)
   - [x] Tests(including UT, IT, E2E) are added to verify the new feature.
     - UTs across the new receiver plugin (engine / cluster / state / apply / rest)
     - E2E under `test/e2e-v2/cases/runtime-rule/`:
       - `mal-storage/{banyandb,elasticsearch,postgresql}` — full 10-phase lifecycle
         (CREATE → FILTER_ONLY → STRUCTURAL → DUMP → 4× ILLEGAL → SHAPE-BREAK →
         INACTIVATE → ACTIVATE → DELETE → DUMP) with a per-phase \`step\` label
       - `lal` — LAL hot-swap with a log-mal aggregation rule, swctl asserts that
         the extracted metric carries the swap-flipped step label
       - `cluster` — 2-OAP convergence over ZooKeeper
   - [ ] If it's UI related, attach the screenshots below.
   
   - [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number.
   - [x] Update the [\`CHANGES\` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).
   
   #### Highlights
   
   - **REST surface** — addOrUpdate / inactivate / delete / get / bundled / list
     / dump on port 17128, raw YAML body for writes, NDJSON / tar.gz for reads.
   - **Cluster-aware** — Suspend/Resume RPC over the cluster bus pauses dispatch
     on every peer for structural cutovers; Forward RPC routes off-main pushes
     through the cluster main; periodic scan (~30 s) self-heals missed updates,
     60 s backstop catches missed Resumes.
   - **Storage-safe** — StorageModels.remove + per-backend dropTable so a runtime
     delete drops the BanyanDB measure (and the model on every backend) without
     restarting. ES/JDBC dropTable is intentionally a no-op (append-only),
     documented per-backend.
   - **BanyanDB schema-watch fence** — schema mutations wait (best-effort,
     bounded 2 s) for every data node to apply the new revision before
     unparking dispatch. Typical case: a 200 OK marks the moment the cluster's
     data boundary moves.
   - **Override SPI** — \`RuntimeRuleOverrideResolver\` lets operator-side overrides
     thread into bundle resolution without rewriting static rule files.
   - **Bundled vs runtime overlay** — at boot, OAP merges bundled rule files with
     persisted runtime rules, so the cluster never silently regresses to
     bundled defaults after a restart.
   
   #### Security
   
   The admin port has no built-in authentication; the module is disabled by
   default. Operators MUST gateway-protect with IP allow-lists, audit every
   request, and keep the port off the cluster-external interface — the security
   notice in \`docs/en/security/README.md\` documents the duty.
   
   #### Dependency bumps
   
   Driven by the new BanyanDB schema-consistency RPCs whose generated validation
   code requires the protobuf-java 4.x runtime:
   
   - gRPC \`1.70.0\` → \`1.80.0\`
   - protobuf-java \`3.25.5\` → \`4.33.1\`
   - pgv (protoc-gen-validate) \`1.2.1\` → \`1.3.0\`
   - Netty \`4.2.10.Final\` → \`4.2.12.Final\`
   - Netty-tcnative \`2.0.75\` → \`2.0.77\`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Runtime rule hot-update for MAL and LAL [skywalking]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng merged PR #13851:
URL: https://github.com/apache/skywalking/pull/13851


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org