You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by wu...@apache.org on 2026/05/07 15:59:16 UTC
(skywalking) 01/01: SWIP-13: Live Debugger for MAL / LAL / OAL + admin-server + runtime-rule integration.
This is an automated email from the ASF dual-hosted git repository.
wu-sheng pushed a commit to branch swip-13-dsl-debugger
in repository https://gitbox.apache.org/repos/asf/skywalking.git
commit 6a11ba3f8c961ad9d458142529543741f8b5c6d5
Author: Wu Sheng <wu...@foxmail.com>
AuthorDate: Thu May 7 22:06:44 2026 +0800
SWIP-13: Live Debugger for MAL / LAL / OAL + admin-server + runtime-rule integration.
Implements [SWIP-13](docs/en/swip/SWIP-13.md) end-to-end and lands two
supporting infrastructure changes the operator-facing API surface depends on:
1. New `admin-server` module — shared host for admin / on-demand write APIs.
Runs on TWO ports: an HTTP REST surface (default 17128) and an
admin-internal gRPC bus (default 17129) carrying peer-to-peer cluster
RPCs (runtime-rule Suspend / Resume / Forward; DSL debug install /
collect / stop / stopByClientId). The admin-internal bus is a dedicated
transport separate from the public agent / cluster gRPC port
(`core.gRPCPort`, default 11800), so privileged admin RPCs stay out of
the agent network's blast radius. Disabled by default
(`SW_ADMIN_SERVER=default`); has no built-in authentication, must be
gateway-protected.
2. Runtime rule hot-update — REST routes (addOrUpdate / inactivate /
delete / get / list / dump) now mount on `admin-server`'s HTTP host.
All admin writes serialize on a single deterministic main OAP per
cluster (sorted-first peer, no leader election); non-main nodes
transparently forward over the admin-internal gRPC bus, so an L7 LB in
front of the admin port can route any operator request to any OAP.
Cluster convergence on the periodic refresh tick is configurable via
`receiver-runtime-rule.refreshRulesPeriod` (default 30 s). The
runtime-rule plugin's old HTTP-server keys
(`receiver-runtime-rule.restHost`/`restPort`/...) are removed; host-
level knobs move under the new `admin-server` block.
3. DSL Debug API (SWIP-13) — new `dsl-debugging` module. Sample-based
runtime debugger that captures per-stage inputs/outputs as MAL/LAL/OAL
process live ingest. Generated bytecode carries one volatile-bool gate
per probe call site; idle path is a single volatile load JIT eliminates
after warm-up. Active sessions fan out to every cluster peer over the
admin-internal gRPC bus so each peer captures its own slice.
- LB-safe routing: any node can serve any verb. POST mints `sessionId`
on the receiving node, broadcasts install to peers, returns
`404 rule_not_found` only when no node owns the rule (404 body
carries `peers[]` so operators see which OAP rejected what). Failed
installs broadcast Stop to clean up any silent-success peer whose
ack timed out. GET returns `404 session_not_found` only when every
node disowns the id.
- Capture surface bounded on two dimensions only — there is NO
per-session byte cap and no structural sub-caps. Hard ceilings:
`MAX_ACTIVE_SESSIONS=200` per node (429 too_many_sessions when full),
`recordCap` ≤ 10000 + `retentionMillis` ≤ 1 h per session
(400 invalid_limits on out-of-range). Total bytes are reported on
every GET response so operators can verify their heap budget.
- LAL granularity flag: `block` (default — parser/extractor/sink) or
`statement` (one `line` entry per extractor statement, with verbatim
DSL slice + source line).
- MAL captures render the full surviving SampleFamily map
(`{"families": N, "items": [...]}`) so multi-metric expressions show
cross-family filter narrowing.
- `sourceText` carries the verbatim ANTLR slice byte-for-byte; `dsl`
and structured `rule` envelope per-record so hot-updates mid-session
don't make captures ambiguous.
- MAL hand-written probe sites null-check the GateHolder so a build
with `injectionEnabled=false` runs MAL analysis without NPEs.
- OAL terminal `appendEmit` produces a TYPE_OUTPUT sample carrying the
L1-ready Metrics snapshot (matches the wire contract).
- OAL debug-source sidecar (.java alongside .class for IDE source-
attach) compiles cleanly: WithMetadata FQCN corrected, final fields
get `null` initializers, stub methods get bodies.
Disabled by default (`SW_DSL_DEBUGGING=default` *and*
`SW_ADMIN_SERVER=default`); `injectionEnabled` defaults `true` once the
module is enabled. Per-DSL operator references under
`docs/en/setup/backend/admin-api/dsl-debugging-{mal,lal,oal}.md`.
End-to-end coverage: four new e2e cases under
`test/e2e-v2/cases/dsl-debugging/{mal,lal-block,lal-statement,oal}` drive
the full session lifecycle against real telemetry through BanyanDB +
runtime-rule and assert the captured wire shape per DSL.
Operators upgrading to 10.5.0 with `SW_RECEIVER_RUNTIME_RULE=default`
must also set `SW_ADMIN_SERVER=default`; OAP fails fast at startup
otherwise.
---
.github/workflows/skywalking.yaml | 8 +
docs/en/changes/changes.md | 61 +-
.../runtime-rule-hot-update.md | 4 +-
docs/en/security/README.md | 62 +-
.../setup/backend/admin-api/dsl-debugging-lal.md | 313 +++
.../setup/backend/admin-api/dsl-debugging-mal.md | 241 ++
.../setup/backend/admin-api/dsl-debugging-oal.md | 236 ++
docs/en/setup/backend/admin-api/dsl-debugging.md | 154 +
docs/en/setup/backend/admin-api/readme.md | 134 +
.../runtime-rule.md} | 44 +-
docs/en/setup/backend/configuration-vocabulary.md | 29 +-
docs/en/swip/SWIP-13.md | 2950 ++++++++++++++++++++
docs/en/swip/readme.md | 4 +-
docs/menu.yml | 18 +-
.../listener/DatabaseSlowStatementBuilder.java | 17 +
.../trace/parser/listener/SampledTraceBuilder.java | 21 +
oap-server/analyzer/log-analyzer/CLAUDE.md | 7 +-
.../log/analyzer/v2/compiler/LALBlockCodegen.java | 7 +
.../analyzer/v2/compiler/LALClassGenerator.java | 132 +-
.../log/analyzer/v2/compiler/LALCodegenHelper.java | 56 +
.../log/analyzer/v2/compiler/LALScriptModel.java | 61 +
.../log/analyzer/v2/compiler/LALScriptParser.java | 47 +-
.../skywalking/oap/log/analyzer/v2/dsl/DSL.java | 26 +
.../oap/log/analyzer/v2/dsl/ExecutionContext.java | 137 +
.../oap/log/analyzer/v2/dsl/LalExpression.java | 21 +-
.../v2/dsl/spec/extractor/MetricExtractor.java | 10 +
.../oap/log/analyzer/v2/dsldebug/LALDebug.java | 89 +
.../log/analyzer/v2/dsldebug/LALDebugRecorder.java | 81 +
.../analyzer/v2/dsldebug/LalPayloadDebugDump.java | 147 +
.../analyzer/v2/dsldebug/LalStaticBindingHook.java | 91 +
.../v2/provider/LogAnalyzerModuleProvider.java | 6 +-
.../provider/log/listener/LogFilterListener.java | 84 +-
.../provider/log/listener/RecordSinkListener.java | 11 +
.../log/analyzer/v2/spi/LALSourceTypeProvider.java | 16 +
.../v2/compiler/LALClassGeneratorTestBase.java | 9 +
oap-server/analyzer/meter-analyzer/CLAUDE.md | 7 +-
.../skywalking/oap/meter/analyzer/v2/Analyzer.java | 71 +
.../oap/meter/analyzer/v2/MetricConvert.java | 39 +-
.../analyzer/v2/compiler/MALBytecodeHelper.java | 33 +
.../analyzer/v2/compiler/MALClassGenerator.java | 105 +-
.../analyzer/v2/compiler/MALCodegenHelper.java | 50 +
.../meter/analyzer/v2/compiler/MALExprCodegen.java | 37 +
.../analyzer/v2/compiler/MALExpressionModel.java | 37 +-
.../v2/compiler/MALMethodChainCodegen.java | 23 +-
.../analyzer/v2/compiler/MALScriptParser.java | 21 +-
.../skywalking/oap/meter/analyzer/v2/dsl/DSL.java | 5 +
.../oap/meter/analyzer/v2/dsl/Expression.java | 11 +
.../meter/analyzer/v2/dsl/FilterExpression.java | 2 +
.../oap/meter/analyzer/v2/dsl/MalExpression.java | 15 +
.../oap/meter/analyzer/v2/dsl/SampleFamily.java | 51 +
.../oap/meter/analyzer/v2/dsldebug/MALDebug.java | 137 +
.../analyzer/v2/dsldebug/MALDebugRecorder.java | 97 +
.../analyzer/v2/dsldebug/MalStaticBindingHook.java | 112 +
.../v2/compiler/MALClassGeneratorTest.java | 11 +
oap-server/oal-rt/CLAUDE.md | 14 +-
.../org/apache/skywalking/oal/v2/OALEngineV2.java | 5 +-
.../skywalking/oal/v2/generator/CodeGenModel.java | 42 +
.../oal/v2/generator/MetricDefinitionEnricher.java | 21 +
.../oal/v2/generator/OALClassGeneratorV2.java | 225 +-
.../skywalking/oal/v2/model/FilterExpression.java | 17 +
.../skywalking/oal/v2/model/FunctionCall.java | 20 +-
.../skywalking/oal/v2/model/MetricDefinition.java | 29 +
.../skywalking/oal/v2/parser/OALListenerV2.java | 43 +
.../dispatcher/doMetricsWithDebug.ftl | 83 +
.../oal/v2/generator/RuntimeOALGenerationTest.java | 11 +-
oap-server/pom.xml | 1 +
oap-server/server-admin/admin-server/pom.xml | 30 +
.../server/cluster/AdminClusterChannelManager.java | 91 +
.../cluster/AdminClusterChannelManagerImpl.java | 266 ++
.../admin/server/module/AdminServerModule.java | 60 +
.../server/module/AdminServerModuleConfig.java | 82 +
.../server/module/AdminServerModuleProvider.java | 242 ++
...ywalking.oap.server.library.module.ModuleDefine | 1 +
...alking.oap.server.library.module.ModuleProvider | 1 +
.../dsl-debugging}/pom.xml | 77 +-
.../dsl/debugging/cluster/ClusterPeerCaller.java | 145 +
.../cluster/DSLDebuggingClusterClient.java | 136 +
.../cluster/DSLDebuggingClusterServiceImpl.java | 200 ++
.../admin/dsl/debugging/cluster/PeerOutcome.java | 54 +
.../dsl/debugging/lal/LALDebugRecorderFactory.java | 49 +
.../dsl/debugging/lal/LALDebugRecorderImpl.java | 137 +
.../admin/dsl/debugging/lal/LALHolderRegistry.java | 72 +
.../dsl/debugging/mal/MALDebugRecorderFactory.java | 50 +
.../dsl/debugging/mal/MALDebugRecorderImpl.java | 210 ++
.../admin/dsl/debugging/mal/MALHolderRegistry.java | 105 +
.../dsl/debugging/module/DSLDebuggingModule.java | 68 +
.../debugging/module/DSLDebuggingModuleConfig.java | 56 +
.../module/DSLDebuggingModuleProvider.java | 279 ++
.../dsl/debugging/oal/OALDebugRecorderFactory.java | 44 +
.../dsl/debugging/oal/OALDebugRecorderImpl.java | 154 +
.../admin/dsl/debugging/oal/OALHolderLookup.java | 61 +
.../dsl/debugging/oal/RuntimeOalRestHandler.java | 231 ++
.../debugging/rest/DSLDebuggingRestHandler.java | 639 +++++
.../debugging/session/AbstractDebugRecorder.java | 245 ++
.../dsl/debugging/session/DebugHolderLookup.java | 58 +
.../debugging/session/DebugRecorderFactory.java | 48 +
.../admin/dsl/debugging/session/DebugSession.java | 49 +
.../debugging/session/DebugSessionRegistry.java | 281 ++
.../dsl/debugging/session/ExecutionRecord.java | 108 +
.../admin/dsl/debugging/session/Granularity.java | 69 +
.../dsl/debugging/session/InstallOutcome.java | 60 +
.../server/admin/dsl/debugging/session/Sample.java | 92 +
.../admin/dsl/debugging/session/SessionLimits.java | 91 +
.../src/main/proto/dsl-debugging-cluster.proto | 209 ++
...ywalking.oap.server.library.module.ModuleDefine | 1 +
...alking.oap.server.library.module.ModuleProvider | 1 +
.../DSLDebuggingClusterServiceImplTest.java | 219 ++
.../session/DebugSessionRegistryTest.java | 254 ++
.../dsl/debugging/session/RecorderCapTest.java | 94 +
.../pom.xml | 32 +-
.../runtime-rule}/pom.xml | 22 +-
.../receiver/runtimerule/apply/DSLDelta.java | 0
.../runtimerule/apply/DeltaClassifier.java | 0
.../receiver/runtimerule/apply/LalFileApplier.java | 0
.../receiver/runtimerule/apply/MalFileApplier.java | 0
.../runtimerule/apply/MalShapeExtractor.java | 0
.../receiver/runtimerule/cluster/MainRouter.java | 116 +
.../cluster/RuntimeRuleClusterClient.java | 68 +-
.../cluster/RuntimeRuleClusterServiceImpl.java | 0
.../receiver/runtimerule/engine/ApplyContext.java | 0
.../receiver/runtimerule/engine/ApplyInputs.java | 0
.../runtimerule/engine/Classification.java | 0
.../receiver/runtimerule/engine/CompiledDSL.java | 0
.../runtimerule/engine/EngineCompileException.java | 0
.../receiver/runtimerule/engine/RuleEngine.java | 0
.../runtimerule/engine/RuleEngineRegistry.java | 0
.../runtimerule/engine/lal/CompiledLalDSL.java | 0
.../runtimerule/engine/lal/LalApplyContext.java | 0
.../runtimerule/engine/lal/LalRuleEngine.java | 98 +-
.../runtimerule/engine/mal/CompiledMalDSL.java | 0
.../runtimerule/engine/mal/MalApplyContext.java | 0
.../runtimerule/engine/mal/MalRuleEngine.java | 95 +-
.../extension/DbOverrideRuntimeRuleResolver.java | 0
.../receiver/runtimerule/metrics/LockMetrics.java | 0
.../runtimerule/module/RuntimeRuleModule.java | 0
.../module/RuntimeRuleModuleConfig.java | 19 +-
.../module/RuntimeRuleModuleProvider.java | 104 +-
.../receiver/runtimerule/reconcile/DSLManager.java | 30 +-
.../runtimerule/reconcile/DSLRuntimeApply.java | 0
.../runtimerule/reconcile/DSLRuntimeDelete.java | 0
.../reconcile/DSLRuntimeUnregister.java | 0
.../runtimerule/reconcile/DSLScriptKey.java | 0
.../runtimerule/reconcile/PendingApplyCommit.java | 0
.../receiver/runtimerule/reconcile/RuleSync.java | 31 +-
.../runtimerule/reconcile/StaticRuleLoader.java | 0
.../reconcile/StructuralCommitCoordinator.java | 0
.../runtimerule/reconcile/SuspendResult.java | 0
.../reconcile/SuspendResumeCoordinator.java | 0
.../receiver/runtimerule/rest/DeleteMode.java | 0
.../runtimerule/rest/RuntimeRuleRestHandler.java | 0
.../runtimerule/rest/RuntimeRuleService.java | 48 +-
.../runtimerule/state/AppliedRuleScript.java | 0
.../runtimerule/state/DSLRuntimeState.java | 0
.../receiver/runtimerule/state/EngineApplied.java | 0
.../receiver/runtimerule/util/ContentHash.java | 0
.../src/main/proto/runtime-rule-cluster.proto | 45 +-
...erver.core.rule.ext.RuntimeRuleOverrideResolver | 0
...ywalking.oap.server.library.module.ModuleDefine | 0
...alking.oap.server.library.module.ModuleProvider | 0
.../meter/analyzer/v2/dsl/TestSampleFamily.java | 0
.../receiver/runtimerule/apply/DSLDeltaTest.java | 0
.../runtimerule/apply/DeltaClassifierTest.java | 0
.../runtimerule/apply/LalFileApplierTest.java | 0
.../runtimerule/apply/MalFileApplierTest.java | 0
.../runtimerule/cluster/MainRouterTest.java | 16 +-
.../runtimerule/rest/GuardrailIntegrationTest.java | 0
.../rest/RuntimeRuleRestHandlerTest.java | 0
.../state/AppliedRuleScriptLockTest.java | 0
.../runtimerule/state/DSLRuntimeStateTest.java | 0
.../receiver/runtimerule/util/ContentHashTest.java | 0
.../server/core/analysis/DispatcherManager.java | 51 +
.../server/core/analysis/metrics/ApdexMetrics.java | 9 +
.../server/core/analysis/metrics/CPMMetrics.java | 7 +
.../server/core/analysis/metrics/CountMetrics.java | 6 +
.../core/analysis/metrics/DoubleAvgMetrics.java | 8 +
.../core/analysis/metrics/LongAvgMetrics.java | 8 +
.../core/analysis/metrics/MaxDoubleMetrics.java | 6 +
.../core/analysis/metrics/MaxLongMetrics.java | 6 +
.../oap/server/core/analysis/metrics/Metrics.java | 41 +-
.../core/analysis/metrics/MinDoubleMetrics.java | 6 +
.../core/analysis/metrics/MinLongMetrics.java | 6 +
.../core/analysis/metrics/PercentMetrics.java | 8 +
.../core/analysis/metrics/PercentileMetrics.java | 7 +
.../server/core/analysis/metrics/RateMetrics.java | 8 +
.../server/core/analysis/metrics/SumMetrics.java | 6 +
.../core/browser/source/BrowserAppPagePerf.java | 28 +
.../core/browser/source/BrowserAppPageTraffic.java | 14 +
.../server/core/browser/source/BrowserAppPerf.java | 26 +
.../browser/source/BrowserAppResourcePerf.java | 18 +
.../source/BrowserAppSingleVersionPerf.java | 28 +
.../source/BrowserAppSingleVersionTraffic.java | 14 +
.../core/browser/source/BrowserAppTraffic.java | 12 +
.../source/BrowserAppWebInteractionPerf.java | 14 +
.../browser/source/BrowserAppWebVitalsPerf.java | 16 +
.../core/browser/source/BrowserErrorLog.java | 16 +
.../oap/server/core/classloader/Catalog.java | 3 +-
.../core/dsldebug/DSLDebugCodegenSwitch.java | 71 +
.../server/core/dsldebug/DebugHolderProvider.java | 54 +
.../oap/server/core/dsldebug/DebugRecorder.java | 62 +
.../oap/server/core/dsldebug/DslContentHash.java} | 21 +-
.../oap/server/core/dsldebug/GateHolder.java | 232 ++
.../oap/server/core/dsldebug/LogDataDebugDump.java | 144 +
.../oap/server/core/dsldebug/OALDebug.java | 88 +
.../oap/server/core/dsldebug/OALDebugRecorder.java | 58 +
.../oap/server/core/dsldebug/RuleKey.java | 60 +
.../oap/server/core/dsldebug/ToJson.java | 64 +
.../instance/InstanceHierarchyRelation.java | 20 +
.../service/ServiceHierarchyRelation.java | 16 +
.../server/core/oal/rt/OALEngineLoaderService.java | 10 +
.../oap/server/core/source/CacheAccess.java | 15 +
.../oap/server/core/source/CacheSlowAccess.java | 19 +
.../oap/server/core/source/CiliumEndpoint.java | 19 +
.../server/core/source/CiliumEndpointRelation.java | 23 +
.../oap/server/core/source/CiliumService.java | 19 +
.../server/core/source/CiliumServiceInstance.java | 20 +
.../core/source/CiliumServiceInstanceRelation.java | 23 +
.../server/core/source/CiliumServiceRelation.java | 21 +
.../oap/server/core/source/DatabaseAccess.java | 14 +
.../server/core/source/DatabaseSlowStatement.java | 16 +
.../core/source/EBPFProcessProfilingSchedule.java | 14 +
.../oap/server/core/source/EBPFProfilingData.java | 15 +
.../oap/server/core/source/Endpoint.java | 26 +
.../oap/server/core/source/EndpointMeta.java | 14 +
.../oap/server/core/source/EndpointRelation.java | 25 +
.../server/core/source/EnvoyInstanceMetric.java | 16 +
.../oap/server/core/source/GenAIModelAccess.java | 19 +
.../server/core/source/GenAIProviderAccess.java | 16 +
.../skywalking/oap/server/core/source/ISource.java | 23 +-
.../oap/server/core/source/JFRProfilingData.java | 14 +
.../oap/server/core/source/K8SEndpoint.java | 17 +
.../oap/server/core/source/K8SService.java | 20 +
.../oap/server/core/source/K8SServiceInstance.java | 16 +
.../core/source/K8SServiceInstanceRelation.java | 18 +
.../oap/server/core/source/K8SServiceRelation.java | 17 +
.../oap/server/core/source/LALOutputBuilder.java | 55 +-
.../skywalking/oap/server/core/source/Log.java | 24 +
.../oap/server/core/source/LogBuilder.java | 122 +-
.../oap/server/core/source/MQAccess.java | 15 +
.../oap/server/core/source/MQEndpointAccess.java | 17 +
.../core/source/NetworkAddressAliasSetup.java | 12 +
.../oap/server/core/source/PprofProfilingData.java | 16 +-
.../skywalking/oap/server/core/source/Process.java | 18 +
.../oap/server/core/source/ProcessRelation.java | 15 +
.../skywalking/oap/server/core/source/Segment.java | 18 +
.../skywalking/oap/server/core/source/Service.java | 26 +
.../core/source/ServiceDatabaseSlowStatement.java | 16 +
.../oap/server/core/source/ServiceInstance.java | 20 +
.../server/core/source/ServiceInstanceCLRCPU.java | 15 +
.../server/core/source/ServiceInstanceCLRGC.java | 18 +
.../core/source/ServiceInstanceCLRThread.java | 18 +
.../server/core/source/ServiceInstanceJVMCPU.java | 15 +
.../core/source/ServiceInstanceJVMClass.java | 17 +
.../server/core/source/ServiceInstanceJVMGC.java | 17 +
.../core/source/ServiceInstanceJVMMemory.java | 19 +
.../core/source/ServiceInstanceJVMMemoryPool.java | 19 +
.../core/source/ServiceInstanceJVMThread.java | 21 +
.../core/source/ServiceInstanceRelation.java | 26 +
.../server/core/source/ServiceInstanceUpdate.java | 11 +
.../oap/server/core/source/ServiceLabel.java | 14 +
.../oap/server/core/source/ServiceMeta.java | 11 +
.../oap/server/core/source/ServiceRelation.java | 29 +
.../oap/server/core/source/TCPService.java | 16 +
.../oap/server/core/source/TCPServiceInstance.java | 16 +
.../core/source/TCPServiceInstanceRelation.java | 21 +
.../core/source/TCPServiceInstanceUpdate.java | 11 +
.../oap/server/core/source/TCPServiceRelation.java | 21 +
.../oap/server/core/source/TagAutocomplete.java | 13 +
.../server/core/zipkin/source/ZipkinService.java | 11 +
.../core/zipkin/source/ZipkinServiceRelation.java | 12 +
.../core/zipkin/source/ZipkinServiceSpan.java | 12 +
.../oap/server/core/zipkin/source/ZipkinSpan.java | 27 +
.../oap/server/core/dsldebug/GateHolderTest.java | 163 ++
.../oap/server/core/dsldebug/RuleKeyTest.java | 49 +
.../envoy/EnvoyHTTPLALSourceTypeProvider.java | 6 +-
.../envoy/persistence/EnvoyAccessLogBuilder.java | 57 +-
.../otlp/OpenTelemetryMetricRequestProcessor.java | 8 +-
oap-server/server-receiver-plugin/pom.xml | 1 -
.../receiver/runtimerule/cluster/MainRouter.java | 87 -
.../provider/handler/TelegrafServiceHandler.java | 8 +-
oap-server/server-starter/pom.xml | 12 +-
.../src/main/resources/application.yml | 93 +-
.../dsl-debugging/lal-block/docker-compose.yml | 60 +
.../dsl-debugging/lal-block/dsl-debug-flow.sh | 172 ++
test/e2e-v2/cases/dsl-debugging/lal-block/e2e.yaml | 66 +
.../cases/dsl-debugging/lal-block/expected/ok.txt | 1 +
.../dsl-debugging/lal-block/log-emitter/Dockerfile | 10 +-
.../dsl-debugging/lal-block/log-emitter/emitter.py | 74 +
.../dsl-debugging/lal-statement/docker-compose.yml | 60 +
.../dsl-debugging/lal-statement/dsl-debug-flow.sh | 174 ++
.../cases/dsl-debugging/lal-statement/e2e.yaml | 65 +
.../dsl-debugging/lal-statement/expected/ok.txt | 1 +
.../lal-statement/log-emitter/Dockerfile | 10 +-
.../lal-statement/log-emitter/emitter.py | 73 +
.../seed-rules/lal-multistatement.yaml | 20 +-
.../cases/dsl-debugging/mal/docker-compose.yml | 78 +
.../cases/dsl-debugging/mal/dsl-debug-flow.sh | 215 ++
test/e2e-v2/cases/dsl-debugging/mal/e2e.yaml | 64 +
.../e2e-v2/cases/dsl-debugging/mal/expected/ok.txt | 1 +
.../mal/seed-rules/mal-with-filter.yaml | 44 +
.../cases/dsl-debugging/oal/docker-compose.yml | 64 +
.../cases/dsl-debugging/oal/dsl-debug-flow.sh | 204 ++
test/e2e-v2/cases/dsl-debugging/oal/e2e.yaml | 69 +
.../e2e-v2/cases/dsl-debugging/oal/expected/ok.txt | 1 +
.../mal-storage/otlp-emitter/emitter.py | 72 +
test/e2e-v2/script/docker-compose/base-compose.yml | 7 +
305 files changed, 16933 insertions(+), 523 deletions(-)
diff --git a/.github/workflows/skywalking.yaml b/.github/workflows/skywalking.yaml
index 784d60eb7c..a680fc3234 100644
--- a/.github/workflows/skywalking.yaml
+++ b/.github/workflows/skywalking.yaml
@@ -407,6 +407,14 @@ jobs:
config: test/e2e-v2/cases/runtime-rule/lal/e2e.yaml
- name: Runtime Rule Cluster Convergence
config: test/e2e-v2/cases/runtime-rule/cluster/e2e.yaml
+ - name: DSL Debug API — MAL
+ config: test/e2e-v2/cases/dsl-debugging/mal/e2e.yaml
+ - name: DSL Debug API — OAL
+ config: test/e2e-v2/cases/dsl-debugging/oal/e2e.yaml
+ - name: DSL Debug API — LAL block
+ config: test/e2e-v2/cases/dsl-debugging/lal-block/e2e.yaml
+ - name: DSL Debug API — LAL statement
+ config: test/e2e-v2/cases/dsl-debugging/lal-statement/e2e.yaml
- name: Alarm ES
config: test/e2e-v2/cases/alarm/es/e2e.yaml
diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md
index e282aff5e3..e4d7f2debd 100644
--- a/docs/en/changes/changes.md
+++ b/docs/en/changes/changes.md
@@ -2,6 +2,26 @@
#### Project
+* **New `admin-server` module — shared host for admin / on-demand write APIs.** Runs on
+ **two ports**: an HTTP REST surface (default `17128`) for operator-facing endpoints,
+ and an **admin-internal gRPC bus** (default `17129`) for peer-to-peer cluster RPCs
+ (runtime-rule Suspend / Resume / Forward; DSL debug install / collect / stop /
+ stopByClientId). The admin-internal bus is a dedicated transport separate from the
+ public agent / cluster gRPC port (`core.gRPCPort`, default `11800`) so privileged
+ admin RPCs stay out of the agent network's blast radius — operators bind `gRPCHost`
+ to a private peer-to-peer interface only. Both the runtime-rule plugin and the new
+ DSL Debug API (below) mount onto this shared host. Disabled by default
+ (`SW_ADMIN_SERVER=default` to enable); has no built-in authentication and **must** be
+ gateway-protected with IP allow-lists, never exposed to the public internet (see the
+ [Admin API security notice](../setup/backend/admin-api/readme.md)). The runtime-rule
+ config block loses its `restHost`/`restPort`/`restContextPath`/`restIdleTimeOut`/
+ `restAcceptQueueSize`/`httpMaxRequestHeaderSize` keys (and the matching
+ `SW_RECEIVER_RUNTIME_RULE_REST_*` env vars); host-level knobs move under the new
+ `admin-server` block (`SW_ADMIN_SERVER_HOST` / `SW_ADMIN_SERVER_PORT` /
+ `SW_ADMIN_SERVER_GRPC_HOST` / `SW_ADMIN_SERVER_GRPC_PORT` /
+ `SW_ADMIN_SERVER_INTERNAL_COMM_TIMEOUT` etc.). Operators upgrading to 10.5.0 with
+ `SW_RECEIVER_RUNTIME_RULE=default` must also set `SW_ADMIN_SERVER=default`; OAP
+ fails fast at startup otherwise.
* **Runtime rule hot-update for MAL and LAL.** Operators can now ship metric (MAL) and log
(LAL) rule changes without restarting OAP. A push to a new admin endpoint persists the rule
to the configured storage backend, and every node in the cluster converges to the new
@@ -36,9 +56,44 @@
as a tar.gz for backup / DR.
Hot-updates survive OAP restart: at boot OAP merges bundled rule files with persisted
runtime rules, so the cluster never silently regresses to the bundled defaults.
- **The endpoint is disabled by default and listens on port `17128` when enabled. It has
- no built-in authentication — operators must gateway-protect it with IP allow-lists and
- never expose it to the public internet.**
+ All admin writes for a runtime-rule cluster serialize on a single "main" OAP
+ (deterministic sorted-first peer, no leader election) — non-main nodes that receive
+ an HTTP write transparently forward it to the main over the admin-internal gRPC bus,
+ so an L7 load balancer in front of the admin port can route any operator request to
+ any OAP. Cluster convergence on the periodic refresh tick is configurable via
+ `receiver-runtime-rule.refreshRulesPeriod` (default `30` s). **The endpoint is
+ disabled by default and listens on port `17128` (HTTP) when enabled. It has no
+ built-in authentication — operators must gateway-protect it with IP allow-lists and
+ never expose it to the public internet.** Routes mount on the new `admin-server`
+ HTTP host, so enabling the runtime-rule API requires both `SW_ADMIN_SERVER=default`
+ and `SW_RECEIVER_RUNTIME_RULE=default`.
+* **Live debugger for MAL / LAL / OAL** — implements [SWIP-13 Live Debugger for MAL / LAL / OAL](../swip/SWIP-13.md).
+ Sample-based runtime debugger that captures per-stage inputs/outputs as the three DSLs
+ process live ingest. Idle-path cost is one volatile-bool read per probe call site that
+ JIT eliminates after warm-up; active sessions fan out to every cluster peer over the
+ admin-internal gRPC bus so each peer captures its own slice. The fan-out is LB-safe:
+ any node can serve any verb (POST mints `sessionId` on the receiving node, broadcasts
+ install to peers, returns `404 rule_not_found` only when no node owns the rule), so an
+ L7 load balancer in front of the admin port routes operator requests freely. Mounts on
+ the shared `admin-server` host (`/dsl-debugging/*` for session control plane,
+ `/runtime/oal/*` for the OAL rule picker). Disabled by default; enable with
+ `SW_DSL_DEBUGGING=default` *and* `SW_ADMIN_SERVER=default`. `injectionEnabled` is a
+ boot-time codegen switch defaulting to `true` — once the module is enabled, probes
+ fire and sessions record samples; set `false` only if the REST surface is wanted but
+ no codegen-side probe overhead is acceptable. Per-session limits enforce hard caps
+ (`recordCap` ≤ 10000, `retentionMillis` ≤ 1 hour) — out-of-range requests return
+ `400 invalid_limits`. LAL sessions accept a per-session `granularity=block|statement`
+ flag — block mode captures the parser/extractor/sink stages; statement mode
+ additionally records one `line` entry per individual extractor statement, carrying
+ the source-line number and verbatim DSL text so the UI can highlight which statement
+ fired. MAL captures render the file-level filter's surviving SampleFamily map
+ (`{"families": N, "items": [...]}`), so multi-metric expressions show cross-family
+ filter narrowing in the captured payload. Capture payloads include raw log bodies and
+ parsed maps — treat the admin port as authenticated infrastructure per the
+ [Admin API security notice](../setup/backend/admin-api/readme.md). Per-DSL operator
+ references: [MAL](../setup/backend/admin-api/dsl-debugging-mal.md),
+ [OAL](../setup/backend/admin-api/dsl-debugging-oal.md),
+ [LAL](../setup/backend/admin-api/dsl-debugging-lal.md).
* **BanyanDB schema mismatches are now visible at boot, not silent.** If BanyanDB already
holds a resource whose shape doesn't match what the current rule declares (e.g., a rule
was edited on disk while OAP was offline), OAP now skips that resource, logs an ERROR
diff --git a/docs/en/concepts-and-designs/runtime-rule-hot-update.md b/docs/en/concepts-and-designs/runtime-rule-hot-update.md
index 135c8a301f..f1901cc6e9 100644
--- a/docs/en/concepts-and-designs/runtime-rule-hot-update.md
+++ b/docs/en/concepts-and-designs/runtime-rule-hot-update.md
@@ -5,7 +5,7 @@ persist in management storage, survive reboots, and propagate across every node
an OAP cluster within a bounded window. This page explains the contract: what the
feature guarantees, how the cluster stays consistent, and what to expect when
something goes wrong. The HTTP surface is documented separately in
-[Runtime Rule Hot-Update API](../setup/backend/backend-runtime-rule-api.md).
+[Runtime Rule Hot-Update API](../setup/backend/admin-api/runtime-rule.md).
## Vocabulary
@@ -391,7 +391,7 @@ the entire diagnostic surface.
- **OAL hot-update** is out of scope (see "Scope" above).
- **Authentication** is not built in. The admin endpoint is disabled by default;
when enabled it must be gateway-protected. See the
- [API doc](../setup/backend/backend-runtime-rule-api.md) for setup guidance.
+ [API doc](../setup/backend/admin-api/runtime-rule.md) for setup guidance.
- **Bulk import.** `/dump` produces a tar.gz for backup, but restore is "extract
one file, POST it to `/addOrUpdate`". There is no single-call cluster import.
- **Rule rollback.** Storage is last-write-wins; there is no automatic
diff --git a/docs/en/security/README.md b/docs/en/security/README.md
index f0cfaf8a7c..67f9d2bbad 100644
--- a/docs/en/security/README.md
+++ b/docs/en/security/README.md
@@ -51,36 +51,46 @@ those fields, causing XSS or even Remote Code Execution (RCE) issues.
For some sensitive environment, consider to limit the telemetry report frequency in case of DoS/DDoS for exposed OAP
and UI services.
-## Runtime Rule Admin Surface (port 17128)
-
-The `skywalking-runtime-rule-receiver-plugin` exposes an HTTP admin API on port 17128 that
-lets operators **add, override, inactivate, and delete MAL/LAL rule files at runtime** without
-restarting OAP. Rules are compiled and loaded into the OAP JVM on the fly. This surface is
-**far more powerful than the telemetry receiver ports** — a request can register new Javassist-
-compiled bytecode, mutate `MeterSystem` state, and drop backend schema (BanyanDB measures).
-
-The module is **disabled by default**. Enabling it (via `SW_RECEIVER_RUNTIME_RULE=default` or
-the YAML selector) opens port 17128 with **no authentication**. This is intentional for now —
-the design goal is a simple admin socket that a gateway / service mesh wraps with the
-operator's existing auth story.
+## Admin API Surface (ports 17128 + 17129)
+
+The `admin-server` module hosts a shared HTTP port (default `17128`) for admin / on-demand
+write APIs. Today this includes the **Runtime Rule Hot-Update API** (add, override, inactivate,
+delete MAL/LAL rule files at runtime — rules are compiled and loaded into the OAP JVM on the
+fly) and the **DSL Debug API** (sampling debugger that captures live MAL/LAL/OAL rule
+executions, including the raw payloads each rule processed). This surface is **far more
+powerful than the telemetry receiver ports** — a request can register new Javassist-compiled
+bytecode, mutate `MeterSystem` state, drop backend schema (BanyanDB measures), or inspect
+operationally-sensitive payloads (raw log bodies, parsed maps, metric source events).
+
+`admin-server` also opens an **admin-internal gRPC bus** (default `17129`) for peer-to-peer
+cluster RPCs between OAP nodes — runtime-rule `Suspend / Resume / Forward`, DSL debug
+`install / collect / stop`. This is intentionally a **separate transport** from the public
+agent / cluster gRPC port (`core.gRPCPort`, default `11800`). An attacker who reaches `11800`
+to submit telemetry MUST NOT be able to invoke privileged admin RPCs.
+
+`admin-server` is **disabled by default**. Enabling it (`SW_ADMIN_SERVER=default` plus the
+relevant feature module's selector) opens both ports with **no built-in authentication**.
+This is intentional for now — the design goal is a simple admin socket that a gateway /
+service mesh wraps with the operator's existing auth story.
Required operator actions when enabling:
-1. **Never expose port 17128 to the public internet.** Bind to a private network interface or
- `localhost` and reach it through an operator-controlled gateway.
-2. **Gateway-protect with IP allow-list + authentication.** Only the operator team should be
- able to reach the endpoint.
-3. **Audit every request.** Rule content is arbitrary YAML that compiles into the OAP JVM —
+1. **Never expose port 17128 (HTTP) to the public internet.** Bind to a private network
+ interface or `localhost` and reach it through an operator-controlled gateway.
+2. **Never expose port 17129 (admin gRPC) to operators or the agent network.** Bind
+ `gRPCHost` to a private peer-to-peer interface reachable ONLY between OAP nodes. This
+ port has the same blast radius as port 17128; do not gateway-publish it.
+3. **Gateway-protect 17128 with IP allow-list + authentication.** Only the operator team
+ should be able to reach the HTTP endpoint.
+4. **Audit every request.** Rule content is arbitrary YAML that compiles into the OAP JVM —
a malicious rule could exfiltrate data, spike resource use, or create metric-name
- collisions. Treat `POST /runtime/rule/*` as equivalent to shell access on the OAP host.
-4. **Keep the port off the cluster-external interface even in cluster mode.** The cluster-
- internal Suspend RPC is registered on the OAP cluster-bus gRPC server (shared with
- RemoteService / HealthCheck) — that is a separate transport from 17128 and follows the
- same security posture as the rest of the cluster bus.
-
-Without these protections an attacker with network reach to port 17128 can execute arbitrary
-code inside the OAP JVM. See `docs/en/setup/backend/backend-runtime-rule-api.md` for the full
-API surface.
+ collisions. Captured DSL-debug payloads include sensitive operational state. Treat
+ `POST /runtime/rule/*` and `POST /dsl-debugging/*` as equivalent to shell access on the
+ OAP host.
+
+Without these protections an attacker with network reach to port 17128 or 17129 can execute
+arbitrary code inside the OAP JVM. See `docs/en/setup/backend/admin-api/readme.md` for the
+full security notice and per-feature API surface.
## Client-Side Monitoring
diff --git a/docs/en/setup/backend/admin-api/dsl-debugging-lal.md b/docs/en/setup/backend/admin-api/dsl-debugging-lal.md
new file mode 100644
index 0000000000..c65987c1f1
--- /dev/null
+++ b/docs/en/setup/backend/admin-api/dsl-debugging-lal.md
@@ -0,0 +1,313 @@
+# DSL Debug API — LAL
+
+> Status: **shipped**. Operator reference for the LAL slice of the DSL Debug
+> API. Design: [SWIP-13](../../../swip/SWIP-13.md). Index of related pages:
+> [DSL Debug API overview](dsl-debugging.md).
+
+## What it captures
+
+A LAL session attaches to one compiled rule (one entry in a LAL YAML's
+`rules:` list). Every log routed through that rule produces one **record**
+in the response; within a record, each probe stage in the rule's pipeline
+appends one **sample**. The wire shape is:
+
+```text
+nodes[]
+ records[]
+ startedAtMs — record boundary timestamp (ms)
+ dsl — verbatim per-rule DSL text
+ rule — { ruleName, sourceLine }
+ samples[]
+ type — input | function | output
+ sourceText — verbatim DSL slice (statement-mode only;
+ empty in block mode)
+ continueOn — true = pipeline continued past this step;
+ false on `abort()` paths
+ payload — see below
+ sourceLine — 1-based line in the rule body (statement-mode
+ only; omitted in block mode)
+```
+
+Sample types and the probes that emit them:
+
+| `type` | Probe | Fired when |
+|------------|----------------------|-----------------------------------------------------------------------------------|
+| `input` | `appendText` | The pipeline begins processing a log — entry-point probe, fires once per record. |
+| `function` | `appendParser` | A `json{}` / `yaml{}` / `text{}` parser block runs (block mode only). |
+| `function` | `appendExtractor` | The `extractor{}` block finishes (block mode synopsis; skipped in statement mode).|
+| `function` | `appendLine` | Per-statement probe — only fires when `granularity=statement`. |
+| `output` | `appendOutputRecord` | A typed log builder reaches `complete()` after the sink kept the record. |
+| `output` | `appendOutputMetric` | The LAL metric extractor produces a SampleFamily for hand-off to MAL. |
+
+### Sample payload — input vs output (LAL split)
+
+The `input` sample (always the first sample on every record) carries the
+**raw input only** — the `LogData` proto for the agent path, or whatever
+typed input the layer's `LALSourceTypeProvider` declares. All subsequent
+samples carry **only the output** — the `LALOutputBuilder`'s accumulated
+state, shaped like the persisted DB row:
+
+```text
+samples[0].payload (input):
+ { aborted, hasParsed, parsedKeys[], input: { LogData fields, body, tags[] } }
+
+samples[≥1].payload (function/output):
+ { aborted, hasParsed, parsedKeys[],
+ output: {
+ type, name, — builder type + short name (e.g. "Log")
+ service, serviceInstance, endpoint,
+ layer, traceId, segmentId, spanId, timestamp,
+ contentType, content, — read directly from input body
+ (no `toLog()` allocation)
+ tags: [ — merged with status:
+ { key, value, status: "original" }, — from input LogData.tags
+ { key, value, status: "lal-added" }, — added by the rule (no key collision)
+ { key, value, status: "lal-override" } — added by the rule, key also in input
+ ]
+ } }
+```
+
+`status` reflects how each tag came to land in the persisted
+`tagsRawData` column: `original` means it was already on the input log;
+`lal-added` means the rule produced a new key; `lal-override` means the
+rule added a tag whose key also exists on the input (the runtime
+concatenates both — they are NOT clean replacements).
+
+When no session is bound, the codegen-emitted probe call sites are single
+volatile-bool reads — idle cost is effectively free.
+
+## Enabling
+
+Two selectors must be enabled — the shared admin HTTP host (`admin-server`)
+and the DSL-debug feature on top of it:
+
+```bash
+SW_ADMIN_SERVER=default
+SW_DSL_DEBUGGING=default
+```
+
+`injectionEnabled` is a **boot-time codegen switch**, default `true` once the
+`dsl-debugging` module is enabled — the LAL generator emits per-rule
+`GateHolder` fields and probe call sites, so debug sessions capture samples.
+Set `false` only if the REST surface is wanted but no codegen-side probe
+overhead is acceptable; with `false` the LAL bytecode is byte-identical to a
+build without SWIP-13. Flipping the flag requires an OAP restart:
+
+```bash
+SW_DSL_DEBUGGING_INJECTION_ENABLED=false # default is true; set false to disable probes
+```
+
+> SECURITY: capture payloads include raw log bodies, parsed maps, and
+> output-builder field values. Treat the admin port as authenticated
+> infrastructure — see
+> [Admin API readme — Security Notice](readme.md#-security-notice).
+
+## Picking the rule key
+
+A LAL rule's key tuple is `(catalog=lal, name=<file>, ruleName=<rule>)`:
+
+| Field | Source |
+|------------|----------------------------------------------------------------------------------------------------------------------------------------------|
+| `catalog` | `lal` |
+| `name` | The rule **file** — for static rules the YAML file name with extension (e.g. `default.yaml`); for runtime-rule entries the runtime-rule `name` (no `.yaml` suffix). |
+| `ruleName` | The `name:` field of the rule entry within the file (one file may declare several rules). |
+
+Examples:
+
+| Source | Install URL |
+|------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------|
+| Shipped `lal/default.yaml`, rule `default` | `?catalog=lal&name=default.yaml&ruleName=default` |
+| Runtime-rule LAL applied as `name=my-extractor`, rule entry `e2e-app-extractor` | `?catalog=lal&name=my-extractor&ruleName=e2e-app-extractor` |
+
+## Granularity
+
+LAL is the only DSL with **per-statement** capture. Operators choose the mode
+at install time:
+
+| `granularity` | Behaviour |
+|----------------|---------------------------------------------------------------------------------------------------------------------------------|
+| `block` (default) | One `input` sample + at most one `function` sample per block (parser / extractor synopsis) + one `output` sample. |
+| `statement` | One `input` sample + one `function` sample **per individual extractor statement** (with verbatim `sourceText` + `sourceLine`) + one `output` sample. |
+
+The codegen call site for `appendLine` is unconditional in the bytecode; the
+recorder short-circuits it in block mode. Statement mode is intended for
+short interactive debugging — capture volume is roughly N× block mode where
+N is the number of statements in the extractor.
+
+Specify the mode either as a query parameter (wins over body) or in the
+install body:
+
+```bash
+# Query param
+curl -X POST '...?catalog=lal&name=...&ruleName=...&clientId=...&granularity=statement'
+
+# Body field
+curl -X POST '...?catalog=lal&name=...&ruleName=...&clientId=...' \
+ -H 'Content-Type: application/json' \
+ -d '{"granularity": "statement"}'
+```
+
+## End-to-end example — block mode
+
+### 1. Open a debug session against the shipped `default` rule
+
+```bash
+curl -s -X POST \
+ 'http://OAP:17128/dsl-debugging/session?catalog=lal&name=default.yaml&ruleName=default&clientId=alice'
+```
+
+### 2. Drive log ingest, then poll
+
+```bash
+curl -s 'http://OAP:17128/dsl-debugging/session/SESSION_ID'
+```
+
+A trimmed slice (one record = one log):
+
+```json
+{
+ "nodes": [{
+ "records": [{
+ "startedAtMs": 1778114804604,
+ "dsl": "filter {\n sink {\n }\n}\n",
+ "rule": { "ruleName": "default", "sourceLine": "3" },
+ "samples": [
+ { "type": "input", "sourceText": "", "continueOn": true,
+ "payload": {
+ "aborted": false, "hasParsed": true, "parsedKeys": [],
+ "input": { "type": "LogData", "service": "demo-svc",
+ "serviceInstance": "demo-1",
+ "timestamp": 1778114804604,
+ "tags": [ {"key":"marker","value":"e2e"} ],
+ "body": { "format": "TEXT", "text": "hello world" } }
+ } },
+ { "type": "output", "sourceText": "", "continueOn": true,
+ "payload": {
+ "aborted": false, "hasParsed": true, "parsedKeys": [],
+ "output": {
+ "type": "LogBuilder", "name": "Log",
+ "service": "demo-svc", "serviceInstance": "demo-1",
+ "endpoint": "", "layer": "",
+ "traceId": "", "segmentId": "", "spanId": 0,
+ "timestamp": 1778114804604,
+ "contentType": "TEXT", "content": "hello world",
+ "tags": [ { "key":"marker","value":"e2e","status":"original" } ]
+ }
+ } }
+ ]
+ }]
+ }]
+}
+```
+
+The `default` rule has no extractor body and no parser block, so the only
+samples are the entry-point `input` and the terminal `output`.
+
+## End-to-end example — statement mode
+
+### 1. Apply a rule with several extractor statements via runtime-rule
+
+```yaml
+# /tmp/lal-multi.yaml
+rules:
+ - name: app-extractor
+ layer: GENERAL
+ dsl: |
+ filter {
+ extractor {
+ tag stage: 'extractor'
+ tag emitter: 'demo'
+ tag rule: 'multi-statement'
+ }
+ sink {
+ }
+ }
+```
+
+```bash
+curl -s -X POST -H 'Content-Type: text/plain' \
+ --data-binary '@/tmp/lal-multi.yaml' \
+ 'http://OAP:17128/runtime/rule/addOrUpdate?catalog=lal&name=lal-multi'
+```
+
+### 2. Open session with statement granularity
+
+```bash
+curl -s -X POST \
+ 'http://OAP:17128/dsl-debugging/session?catalog=lal&name=lal-multi&ruleName=app-extractor&clientId=alice&granularity=statement'
+```
+
+### 3. Poll — function samples appear, one per `tag` statement, with sourceLine
+
+```json
+{
+ "nodes": [{
+ "records": [{
+ "samples": [
+ { "type": "input", "sourceText": "", "sourceLine": 0,
+ "payload": { /* input LogData */ } },
+ { "type": "function", "sourceText": "tag stage: 'extractor'",
+ "sourceLine": 5,
+ "payload": { /* output snapshot — 1 lalTag added */ } },
+ { "type": "function", "sourceText": "tag emitter: 'demo'",
+ "sourceLine": 6,
+ "payload": { /* output snapshot — 2 lalTags */ } },
+ { "type": "function", "sourceText": "tag rule: 'multi-statement'",
+ "sourceLine": 7,
+ "payload": { /* output snapshot — 3 lalTags */ } },
+ { "type": "output", "sourceText": "",
+ "payload": { /* final builder snapshot */ } }
+ ]
+ }]
+ }]
+}
+```
+
+Each `function` sample's `sourceLine` (1-based, relative to the DSL block)
+and verbatim `sourceText` (ANTLR Interval slice of the
+`ExtractorStatementContext`) identify the exact statement that fired.
+
+### 4. Stop
+
+```bash
+curl -s -X POST 'http://OAP:17128/dsl-debugging/session/SESSION_ID/stop'
+```
+
+## Cluster behaviour
+
+- **Install** broadcasts to every reachable peer; each peer binds its own
+ recorder on its own holder.
+- **Collect** broadcasts and concatenates per-node slices.
+- **Stop** broadcasts; missed acks fall out via retention timeout.
+
+No cross-node merge — each peer's slice is self-contained.
+
+## Failure modes
+
+| Response | Meaning |
+|------------------------------|-------------------------------------------------------------------------|
+| `400 invalid_catalog` | Catalog must be `lal`. |
+| `400 missing_param` | `name` or `ruleName` is missing. |
+| `404 rule_not_found` | No live LAL artifact for the tuple on this node — typo in the file/rule name, or the rule is inactivated. |
+| `503 injection_disabled` | `injectionEnabled=false`. Restart with the flag on to debug. |
+
+## Limits
+
+| Field | Default | Purpose |
+|-------------------|---------------|--------------------------------------------------|
+| `recordCap` | `1000` | Max records before the recorder refuses appends. |
+| `retentionMillis` | `300000` (5m) | Wall-clock retention. |
+| `granularity` | `block` | `block` or `statement` (LAL only). |
+
+Override per-session in the install body:
+
+```json
+{ "recordCap": 200, "retentionMillis": 600000, "granularity": "statement" }
+```
+
+## See also
+
+- [DSL Debug API — MAL](dsl-debugging-mal.md)
+- [DSL Debug API — OAL](dsl-debugging-oal.md)
+- [Runtime Rule Hot-Update API](runtime-rule.md) — apply / inactivate / delete LAL rules.
+- [SWIP-13](../../../swip/SWIP-13.md) — full design.
diff --git a/docs/en/setup/backend/admin-api/dsl-debugging-mal.md b/docs/en/setup/backend/admin-api/dsl-debugging-mal.md
new file mode 100644
index 0000000000..d0200485d2
--- /dev/null
+++ b/docs/en/setup/backend/admin-api/dsl-debugging-mal.md
@@ -0,0 +1,241 @@
+# DSL Debug API — MAL
+
+> Status: **shipped**. Operator reference for the MAL slice of the DSL Debug
+> API. Design: [SWIP-13](../../../swip/SWIP-13.md). Index of related pages:
+> [DSL Debug API overview](dsl-debugging.md).
+
+## What it captures
+
+A MAL session attaches to one metric rule. Every scrape window that survives
+the rule's file-level filter produces one **record** in the response;
+within a record, each probe stage the expression executes appends one
+**sample**. The wire shape is:
+
+```text
+nodes[]
+ records[]
+ startedAtMs — record boundary timestamp (ms)
+ dsl — verbatim per-rule DSL text
+ rule — rule envelope:
+ metricPrefix
+ name — per-rule name (no prefix)
+ filter — file-level filter closure body, if any
+ exp — `exp:` body verbatim
+ expSuffix — file-level expSuffix verbatim, if any
+ samples[]
+ type — input | filter | function | output
+ sourceText — verbatim DSL fragment for this probe
+ continueOn — true (MAL captures kept-only; see overview)
+ payload — SampleFamily.toJson() at this probe stage
+ sourceLine — omitted for MAL (no per-line mapping)
+```
+
+Sample types and the probes that emit them:
+
+| `type` | Probe | Fired when |
+|------------|----------------------|----------------------------------------------------------------------------|
+| `filter` | `captureFilter` | The file-level `filter:` closure runs over the input samples (kept-only). |
+| `input` | `captureInput` | The metric reference at the head of the expression resolves a SampleFamily. |
+| `function` | `captureStage` | An in-expression chain op runs (`sum`, `tagEqual`, `service`, etc.). |
+| `function` | `captureDownsample` | A downsampling op runs (e.g. `rate("PT1M")`). |
+| `output` | `captureMeterEmit` | The metric is emitted to the persistence pipeline (terminal). |
+
+`sample.sourceText` is the **verbatim ANTLR slice** of the chain segment
+from the original `exp:` body — operators can grep the captured text
+against the source byte-for-byte. There is no leading `.` (the dot is
+part of the chain context, not the `MethodCallContext` slice).
+
+`sample.payload` is the structured `SampleFamily.toJson()` at that
+probe stage — every sample's name, label set, value, and timestamp is
+present; truncated at `maxSamplesPerCapture` (default 64) with a
+`+N more` summary.
+
+When no session is bound, the codegen-emitted probe call sites are single
+volatile-bool reads that JIT eliminates after warm-up — idle cost is
+effectively free.
+
+## Enabling
+
+Two selectors must be enabled — the shared admin HTTP host (`admin-server`)
+and the DSL-debug feature on top of it:
+
+```bash
+SW_ADMIN_SERVER=default
+SW_DSL_DEBUGGING=default
+```
+
+`injectionEnabled` is a **boot-time codegen switch**, default `true` once the
+`dsl-debugging` module is enabled — the MAL generator emits per-rule
+`GateHolder` fields and probe call sites, so debug sessions actually capture
+samples. Set `false` only if the REST surface is wanted but no codegen-side
+probe overhead is acceptable; with `false` the MAL bytecode is byte-identical
+to a build without SWIP-13, and `POST /dsl-debugging/session` returns
+`503 injection_disabled`. Flipping the flag requires an OAP restart:
+
+```bash
+SW_DSL_DEBUGGING_INJECTION_ENABLED=false # default is true; set false to disable probes
+```
+
+> SECURITY: capture payloads include MAL builder state and sample-family
+> contents. Treat the admin port as authenticated infrastructure — see
+> [Admin API readme — Security Notice](readme.md#-security-notice).
+
+## Picking the rule key
+
+A session targets one MAL metric rule. The key tuple is
+`(catalog, name, ruleName)`:
+
+| Field | Source |
+|------------|----------------------------------------------------------------------------------------------|
+| `catalog` | One of `otel-rules`, `log-mal-rules`, `telegraf-rules` — the directory the rule file lives in |
+| `name` | The rule **file** name, without `.yaml` |
+| `ruleName` | The full metric name (`metricPrefix` + `_` + per-rule `name`) |
+
+Example — the shipped `otel-rules/vm.yaml` declares a metric prefix
+`vm` and per-rule name `cpu_total_percentage`. The full metric name is
+`vm_cpu_total_percentage`. The session install call:
+
+```
+POST /dsl-debugging/session?catalog=otel-rules&name=vm&ruleName=vm_cpu_total_percentage
+```
+
+To list the metrics a runtime-rule MAL file exposes, query
+`GET /runtime/rule/list` and pull the `ruleName`s associated with the
+catalog/name pair (the runtime-rule receiver records every rule's metric
+catalog).
+
+## End-to-end example
+
+The example uses a runtime-rule-applied MAL rule with a top-level filter
+clause so all probe stages (`filter` → `input` → `function` → `output`)
+appear in the captures.
+
+### 1. Apply the rule
+
+```yaml
+# /tmp/mal-with-filter.yaml
+filter: "{ tags -> tags.service_name == 'my-svc' }"
+metricPrefix: e2e_demo
+expSuffix: service(['service_name'], Layer.GENERAL)
+metricsRules:
+ - name: filtered_requests
+ exp: e2e_demo_request_count_total.sum(['service_name'])
+```
+
+```bash
+curl -s -X POST -H 'Content-Type: text/plain' \
+ --data-binary '@/tmp/mal-with-filter.yaml' \
+ 'http://OAP:17128/runtime/rule/addOrUpdate?catalog=otel-rules&name=mal-with-filter'
+```
+
+### 2. Open a debug session
+
+```bash
+curl -s -X POST \
+ 'http://OAP:17128/dsl-debugging/session?catalog=otel-rules&name=mal-with-filter&ruleName=e2e_demo_filtered_requests&clientId=alice'
+```
+
+### 3. Drive ingest, then poll
+
+```bash
+curl -s 'http://OAP:17128/dsl-debugging/session/SESSION_ID'
+```
+
+A trimmed slice (one record = one scrape window):
+
+```json
+{
+ "sessionId": "76b3266a-...",
+ "capturedAt": 1777967923700,
+ "ruleKey": { "catalog": "otel-rules", "name": "mal-with-filter",
+ "ruleName": "e2e_demo_filtered_requests" },
+ "nodes": [{
+ "nodeId": "0.0.0.0_11800",
+ "status": "ok",
+ "records": [{
+ "startedAtMs": 1777967921000,
+ "dsl": "(e2e_demo_request_count_total.sum(['service_name'])).service(['service_name'], Layer.GENERAL)",
+ "rule": {
+ "metricPrefix": "e2e_demo",
+ "name": "filtered_requests",
+ "filter": "{ tags -> tags.service_name == 'my-svc' }",
+ "exp": "e2e_demo_request_count_total.sum(['service_name'])",
+ "expSuffix": "service(['service_name'], Layer.GENERAL)"
+ },
+ "samples": [
+ { "type": "filter",
+ "sourceText": "{ tags -> tags.service_name == 'my-svc' }",
+ "continueOn": true,
+ "payload": {
+ "families": 1,
+ "items": [ /* one entry per surviving SampleFamily — name, samples count, items[] */ ]
+ } },
+ { "type": "input",
+ "sourceText": "e2e_demo_request_count_total",
+ "continueOn": true,
+ "payload": { /* head SampleFamily — name, samples, items[] */ } },
+ { "type": "function",
+ "sourceText": "sum(['service_name'])",
+ "continueOn": true,
+ "payload": { /* SampleFamily after sum */ } },
+ { "type": "output",
+ "sourceText": "e2e_demo_filtered_requests",
+ "continueOn": true,
+ "payload": { /* terminal meter sample — metric, entity, value, timeBucket */ } }
+ ]
+ }]
+ }]
+}
+```
+
+`sample.sourceText` is the verbatim ANTLR slice — match it against the
+`exp:` body byte-for-byte. The record-level `rule` envelope echoes the
+structured rule config so operators don't have to re-resolve the file.
+
+### 4. Stop
+
+```bash
+curl -s -X POST 'http://OAP:17128/dsl-debugging/session/SESSION_ID/stop'
+```
+
+## Cluster behaviour
+
+- **Install** broadcasts to every reachable peer; each peer binds its own
+ recorder on its own holder so the slice reflects local L1 parsing.
+- **Collect** broadcasts and concatenates per-node slices into `nodes[]`;
+ unreachable peers appear as `status: "unreachable"` rather than being
+ omitted.
+- **Stop** broadcasts; missed acks fall out via per-node retention timeout
+ (default 5 minutes).
+
+No cross-node merge — each slice is self-contained.
+
+## Failure modes
+
+| Response | Meaning |
+|---------------------------|------------------------------------------------------------------------------------|
+| `400 invalid_catalog` | The wire `catalog` is not one of the MAL catalogs. |
+| `400 missing_param` | `name` or `ruleName` is missing. |
+| `404 rule_not_found` | No live MAL artifact for the tuple on this node — rule never loaded, was inactivated, or this node hasn't compiled it yet. |
+| `503 injection_disabled` | `injectionEnabled=false`. Restart with the flag on to debug. |
+| `500 registry_misconfigured` | A recorder factory wiring bug — file an issue. |
+
+## Limits
+
+| Field | Default | Purpose |
+|------------------------|-------------|-------------------------------------------------------------------------------|
+| `recordCap` | `1000` | Max records before the recorder marks itself `captured` and refuses appends. |
+| `retentionMillis` | `300000` (5m) | Wall-clock retention; the session is reaped after the deadline whether or not it was explicitly stopped. |
+
+Override per-session in the `POST /dsl-debugging/session` body:
+
+```json
+{ "recordCap": 200, "retentionMillis": 600000 }
+```
+
+## See also
+
+- [DSL Debug API — OAL](dsl-debugging-oal.md)
+- [DSL Debug API — LAL](dsl-debugging-lal.md)
+- [Runtime Rule Hot-Update API](runtime-rule.md) — apply / inactivate / delete MAL rules.
+- [SWIP-13](../../../swip/SWIP-13.md) — full design.
diff --git a/docs/en/setup/backend/admin-api/dsl-debugging-oal.md b/docs/en/setup/backend/admin-api/dsl-debugging-oal.md
new file mode 100644
index 0000000000..bf82ebfc63
--- /dev/null
+++ b/docs/en/setup/backend/admin-api/dsl-debugging-oal.md
@@ -0,0 +1,236 @@
+# DSL Debug API — OAL
+
+> Status: **shipped**. Operator reference for the OAL slice of the DSL Debug
+> API. Design: [SWIP-13](../../../swip/SWIP-13.md). Index of related pages:
+> [DSL Debug API overview](dsl-debugging.md).
+
+## What it captures
+
+OAL's gate is **per-metric**: each generated metric (e.g.
+`service_relation_server_cpm`) has its own gate holder, and a debug
+session attaches to one metric. Sibling rules under the same source
+dispatcher stay silent.
+
+A session against `(catalog=oal, name=<file>, ruleName=<metric>)`
+captures every source event walked through that one metric's pipeline.
+Every event produces one **record**; within a record, each probe stage
+appends one **sample**:
+
+```text
+nodes[]
+ records[]
+ startedAtMs — record boundary timestamp (ms)
+ dsl — verbatim per-metric `.oal` line
+ rule — { ruleName, sourceLine }
+ samples[]
+ type — input | filter | aggregation | output
+ sourceText — verbatim ANTLR slice from the `.oal` file
+ continueOn — true = pipeline continued past this step
+ payload — { type, scope, fields | timeBucket | … }
+ sourceLine — 1-based line in the source `.oal` file
+```
+
+Sample types and the probes that emit them:
+
+| `type` | Probe | Fired when |
+|---------------|----------------------|----------------------------------------------------------------------------------------------------|
+| `input` | `captureSource` | A source event arrives at the metric's pipeline. Payload = `ISource.toJson()` with all source columns. |
+| `filter` | `captureFilter` | An OAL `.filter(...)` clause runs. **Both** kept (`continueOn=true`) **and** rejected (`continueOn=false`) branches are captured. |
+| `aggregation` | `captureAggregation` | The metric's aggregation function runs (`cpm()`, `percentile2(10)`, ...). Carries the post-aggregation source view. |
+| `output` | `captureEmit` | The metric is emitted to the persistence pipeline (terminal). Payload = `Metrics.toJson()` with `count` / `total` / `value` / `timeBucket` etc. |
+
+### sourceText is the verbatim ANTLR slice
+
+Pulled at `.oal` parse time via
+`ctx.getStart().getInputStream().getText(Interval.of(...))`. Whitespace
+and identifier spelling are byte-identical to the source file:
+
+- `input.sourceText` — the source clause (e.g. `from(ServiceRelation.*)`).
+- `filter.sourceText` — the filter clause **including** the leading `.`
+ (the dot is part of the slice, e.g. `.filter(detectPoint == DetectPoint.SERVER)`).
+- `aggregation.sourceText` — the aggregate function call
+ (e.g. `cpm()` / `percentile2(10)`).
+- `output.sourceText` — the metric name.
+
+Operators can grep the captured `sourceText` against the original `.oal`
+file directly.
+
+### Both kept and rejected filter branches
+
+Unlike MAL (where rejected executions are dropped to avoid tag-cardinality
+noise), OAL captures both filter branches. OAL filters are deterministic
+discriminators — CLIENT-vs-SERVER, layer matchers, status / latency
+predicates — and seeing the rejected source samples (`continueOn=false`)
+is the filter doing its job in plain view, useful for verifying partition
+logic.
+
+When no session is bound, the codegen-emitted probe call sites are single
+volatile-bool reads — idle cost is effectively free.
+
+## Enabling
+
+Two selectors must be enabled — the shared admin HTTP host (`admin-server`)
+and the DSL-debug feature on top of it:
+
+```bash
+SW_ADMIN_SERVER=default
+SW_DSL_DEBUGGING=default
+```
+
+`injectionEnabled` is a **boot-time codegen switch**, default `true` once the
+`dsl-debugging` module is enabled — the OAL dispatcher template emits
+per-metric `GateHolder` fields and probe call sites, so debug sessions
+capture samples. Set `false` only if the REST surface is wanted but no
+codegen-side probe overhead is acceptable; with `false` the OAL bytecode is
+byte-identical to a build without SWIP-13. Flipping the flag requires an
+OAP restart:
+
+```bash
+SW_DSL_DEBUGGING_INJECTION_ENABLED=false # default is true; set false to disable probes
+```
+
+> SECURITY: capture payloads include source-event contents (service names,
+> endpoint names, span attributes). Treat the admin port as authenticated
+> infrastructure — see
+> [Admin API readme — Security Notice](readme.md#-security-notice).
+
+## Picking the rule key
+
+A session targets one OAL **metric**. The key tuple is
+`(catalog=oal, name=<file>, ruleName=<metric>)`:
+
+| Field | Source |
+|------------|-----------------------------------------------------------------------|
+| `catalog` | `oal` |
+| `name` | The `.oal` file the metric is declared in (e.g. `core.oal`). |
+| `ruleName` | The metric name on the LHS of the `=` (e.g. `service_relation_server_cpm`). |
+
+To list the metrics loaded on a node, query
+`GET /runtime/oal/files` — each file's response lists its metrics. The
+same endpoint also exposes the rules registered against each source —
+useful when picking a metric whose filter clauses you want to inspect.
+
+## End-to-end example
+
+The shipped `core.oal` declares `service_relation_server_cpm` against
+`ServiceRelation` with a SERVER detect-point filter:
+
+```
+service_relation_server_cpm = from(ServiceRelation.*)
+ .filter(detectPoint == DetectPoint.SERVER).cpm();
+```
+
+### 1. Open a debug session
+
+```bash
+curl -s -X POST \
+ 'http://OAP:17128/dsl-debugging/session?catalog=oal&name=core.oal&ruleName=service_relation_server_cpm&clientId=alice'
+```
+
+### 2. Drive ingest
+
+Send any agent traffic that produces inter-service spans (HTTP between
+two SkyWalking-instrumented services, gRPC, etc.). The dispatcher fires
+the metric's pipeline on every `ServiceRelation` source event.
+
+### 3. Poll
+
+```bash
+curl -s 'http://OAP:17128/dsl-debugging/session/SESSION_ID'
+```
+
+A trimmed slice (one record = one source event):
+
+```json
+{
+ "ruleKey": { "catalog": "oal", "name": "core.oal",
+ "ruleName": "service_relation_server_cpm" },
+ "nodes": [{
+ "nodeId": "0.0.0.0_11800",
+ "status": "ok",
+ "records": [{
+ "startedAtMs": 1778115085149,
+ "dsl": "service_relation_server_cpm = from(ServiceRelation.*).filter(detectPoint == DetectPoint.SERVER).cpm();",
+ "rule": { "ruleName": "service_relation_server_cpm", "sourceLine": "30" },
+ "samples": [
+ { "type": "input",
+ "sourceText": "from(ServiceRelation.*)",
+ "continueOn": true,
+ "payload": {
+ "type": "ServiceRelation", "scope": 4,
+ "fields": {
+ "sourceServiceName": "e2e-service-consumer",
+ "destServiceName": "e2e-service-provider",
+ "detectPoint": "SERVER",
+ "endpoint": "POST:/users",
+ "componentId": 1, "latency": 962, "status": true,
+ "httpResponseStatusCode": 200,
+ "timeBucket": 202605070051
+ }
+ },
+ "sourceLine": 30 },
+ { "type": "filter",
+ "sourceText": ".filter(detectPoint == DetectPoint.SERVER)",
+ "continueOn": true,
+ "payload": { "type": "ServiceRelation", "fields": { /* same row */ } },
+ "sourceLine": 30 },
+ { "type": "aggregation",
+ "sourceText": "cpm()",
+ "continueOn": true,
+ "payload": { "type": "ServiceRelationServerCpmMetrics",
+ "timeBucket": 202605070051,
+ "count": 1, "total": 1, "value": 1 },
+ "sourceLine": 30 }
+ ]
+ }]
+ }]
+}
+```
+
+A single source event commonly produces a rejected `filter` sample on
+sibling rules (e.g. the CLIENT-detect-point sibling). The rejected
+sample shows `continueOn=false` and no `aggregation`/`output` follows —
+the metric's pipeline stopped at the filter.
+
+### 4. Stop
+
+```bash
+curl -s -X POST 'http://OAP:17128/dsl-debugging/session/SESSION_ID/stop'
+```
+
+## Cluster behaviour
+
+- **Install** broadcasts to every reachable peer; each peer attaches its own
+ recorder to its dispatcher.
+- **Collect** broadcasts and concatenates per-node slices.
+- **Stop** broadcasts; missed acks fall out via retention timeout.
+
+No cross-node merge — each peer's slice is self-contained.
+
+## Failure modes
+
+| Response | Meaning |
+|------------------------------|---------------------------------------------------------------------------|
+| `400 invalid_catalog` | Catalog must be `oal`. |
+| `400 missing_param` | `name` or `ruleName` is missing. |
+| `404 rule_not_found` | No metric for `(name, ruleName)` on this node — typo, or no `.oal` rule loaded. |
+| `503 injection_disabled` | `injectionEnabled=false`. Restart with the flag on to debug. |
+
+## Limits
+
+| Field | Default | Purpose |
+|-------------------|---------------|--------------------------------------------------------|
+| `recordCap` | `1000` | Max records before the recorder refuses appends. |
+| `retentionMillis` | `300000` (5m) | Wall-clock retention. |
+
+Override per-session in the install body:
+
+```json
+{ "recordCap": 200, "retentionMillis": 600000 }
+```
+
+## See also
+
+- [DSL Debug API — MAL](dsl-debugging-mal.md)
+- [DSL Debug API — LAL](dsl-debugging-lal.md)
+- [SWIP-13](../../../swip/SWIP-13.md) — full design.
diff --git a/docs/en/setup/backend/admin-api/dsl-debugging.md b/docs/en/setup/backend/admin-api/dsl-debugging.md
new file mode 100644
index 0000000000..290d912c3d
--- /dev/null
+++ b/docs/en/setup/backend/admin-api/dsl-debugging.md
@@ -0,0 +1,154 @@
+# DSL Debug API
+
+> Status: **shipped**. Sample-based runtime debugger for the three DSLs that
+> drive SkyWalking analysis: **MAL** (meter analysis language), **LAL** (log
+> analysis language), and **OAL** (observability analysis language). Design:
+> [SWIP-13](../../../swip/SWIP-13.md).
+
+## Per-DSL operator references
+
+Each DSL has its own probe surface, payload shape, and rule-key conventions
+— pick the page that matches the rule you're debugging:
+
+- **[MAL](dsl-debugging-mal.md)** — meter rules under `otel-rules`, `log-mal-rules`, `telegraf-rules`. Each captured record is one `SampleFamily` walking through the rule end-to-end (filter → chain ops → meterEmit). Sample payloads carry the complete `SampleFamily` (every sample's name + labels + value + timestamp).
+- **[OAL](dsl-debugging-oal.md)** — per-metric dispatcher capture under `catalog=oal`. Each record is one `ISource` walking through (source entry → filter clauses → aggregation function → emit). Source samples carry the rich `ServiceRelation`-style payload (sourceServiceName, destServiceName, layers, latency, status, detectPoint, ...).
+- **[LAL](dsl-debugging-lal.md)** — log analysis under `catalog=lal`. Each record is one log walking through (text → parser → extractor statements → sink). `granularity=statement` emits one sample per extractor statement; `granularity=block` (default) collapses extractor into one sample.
+
+## Kept-only capture for MAL (filter is the relevance gate)
+
+**MAL** is the only DSL where rejected executions are dropped from
+`records[]`. The reason is tag-cardinality noise: MAL filters
+discriminate on tag values (`service_name == 'foo'`,
+`kind == 'http_server_request_duration'`) and in a multi-tenant or
+multi-component flow, a single rule may reject 99% of the traffic
+routed to it. If every rejection landed as a record, the default
+`recordCap` (32) would fill with noise within seconds and the capture
+would never reach a row demonstrating the rule's actual processing.
+
+So for MAL the contract is: **every record represents one
+SampleFamily that passed the rule's filter clause and walked through
+to `meterEmit`**. The first sample is the input that the filter
+accepted; subsequent samples follow the chain ops; the last is the
+emit.
+
+**OAL** captures both kept and rejected executions. OAL filters are
+deterministic discriminators (CLIENT-vs-SERVER, layer matchers,
+status / latency predicates), not high-cardinality tag noise — seeing
+the rejected-source samples (`continueOn=false`) is the filter doing
+its job in plain view, useful for verifying partition logic.
+
+**LAL** has no analogue (`abort()` is a per-statement short-circuit,
+not a filter); aborted logs publish their accumulated samples with
+`continueOn=false` on the abort point so operators can see where the
+rule gave up.
+
+For MAL operators debugging "why isn't my data being aggregated?":
+the runtime-rule list endpoint surfaces the rule's filter source;
+combine with knowledge of the input tags to identify the mismatch.
+The debug API's job is to show what the rule did with traffic it
+accepted.
+
+## Common ground
+
+What the three DSLs share:
+
+- **One REST surface**: routes mount under `/dsl-debugging/*` on the
+ `admin-server` HTTP host (default port `17128`). The shape of session
+ lifecycle (install, poll, stop) is identical across DSLs — only the
+ rule-key tuple and payload schema differ.
+- **One enable contract**: requires `SW_ADMIN_SERVER=default` and
+ `SW_DSL_DEBUGGING=default`. Probe insertion is gated by the boot-time
+ switch `SW_DSL_DEBUGGING_INJECTION_ENABLED` (default `true` — once the
+ module is enabled, probes fire and sessions record samples). Set this
+ to `false` only if the REST surface is wanted but probe overhead is
+ not; flipping requires an OAP restart.
+- **One cluster contract**: install / collect / stop broadcast to every
+ reachable peer over the **admin-internal gRPC bus** (default port
+ `17129`, owned by `admin-server`) — NOT the public agent / cluster
+ gRPC port (`core.gRPCPort`, default `11800`). Privileged admin RPCs
+ stay off the agent network; operators bind `17129` to a private
+ peer-to-peer interface only. Each peer's slice is self-contained — no
+ L2 join.
+- **One retention contract**: per-session `recordCap` (default `1000`,
+ hard cap `10000`) and `retentionMillis` (default 5 minutes, hard cap
+ 1 hour). Requests outside these bounds return `400 invalid_limits`.
+ A missed `stop` self-cleans on retention timeout.
+- **LB-safe routing**: any node can serve any verb. POST mints the
+ `sessionId` on the receiving node, broadcasts install to all peers,
+ and reports per-peer install state in `peers[]`. GET/stop on a
+ different node aggregate the cluster slices for the same id. So an
+ L7 LB in front of the admin port (default port `17128`) can route
+ any request to any OAP, and the cluster fan-out hides which node
+ actually owns the rule.
+- **One timestamp per slice**: the GET response envelope carries a
+ single `capturedAt` (millis) stamped at GET time. Per-record stamps
+ were redundant — probes fire at clock-tick speed and intra-slice
+ ordering is preserved by array order.
+- **`sourceText` is the verbatim ANTLR slice**: pulled via
+ `ctx.getStart().getInputStream().getText(Interval.of(start, stop))` at
+ parse time, so whitespace and identifier spelling are byte-identical
+ to the original `.yaml` / `.oal`. This applies to MAL chain segments,
+ OAL filter clauses, OAL aggregation calls, and LAL extractor
+ statements. Operators can grep the captured `sourceText` against the
+ source file directly.
+
+Each DSL's page has its own walkthrough — start there for a copy-paste
+example targeting that DSL's rules.
+
+## Session lifecycle
+
+| Verb | Path | Purpose |
+|------|---------------------------------------------------------------------------------------------|------------------------------------------|
+| POST | `/dsl-debugging/session?catalog=&name=&ruleName=&clientId=&granularity=` | Start a debug capture session. |
+| GET | `/dsl-debugging/session/{id}` | Snapshot the captured records. |
+| POST | `/dsl-debugging/session/{id}/stop` | Stop a session (idempotent). |
+| GET | `/dsl-debugging/sessions` | List active sessions on this node. |
+| GET | `/dsl-debugging/status` | Module posture (phase, injectionEnabled). |
+
+`granularity` is LAL-only; MAL/OAL recorders ignore it. Body fields
+(`recordCap`, `retentionMillis`, `granularity`) are optional; defaults from
+the table above apply.
+
+## OAL rule picker (read-only)
+
+The DSL debug API also exposes a read-only listing of the OAL rules
+loaded by an OAP node, used by the debugger UI's rule picker:
+
+| Verb | Path | Purpose |
+|------|---------------------------------|------------------------------------------------------|
+| GET | `/runtime/oal/files` | List the OAL files this OAP loaded at boot, with their sources and registered rules. |
+| GET | `/runtime/oal/files/{name}` | Raw OAL content (UTF-8 text, classpath-bound). |
+| GET | `/runtime/oal/rules/{source}` | Rules registered against a single source (debug-key resolver). |
+
+These do not require `injectionEnabled=true` — they are pure metadata.
+
+## Failure modes
+
+Common across DSLs. Per-DSL pages list any DSL-specific failures.
+
+| Response | Meaning |
+|------------------------------|------------------------------------------------------------------------------------|
+| `400 invalid_catalog` | Wire `catalog` doesn't match any known DSL. |
+| `400 missing_param` | `name`, `ruleName`, or `clientId` query param missing. `clientId` is required so the receiving node can clean up the prior session bound to it before installing a new one. |
+| `400 invalid_limits` | `recordCap` / `retentionMillis` outside the per-session bounds (see above). |
+| `429 too_many_sessions` | Active-session ceiling on the receiving node (200) is full. Stop a session or wait for retention. |
+| `404 rule_not_found` | No live DSL artifact for `(catalog, name, ruleName)` on **any reachable** OAP node — every node and peer reported `NOT_LOCAL`. The 404 body carries `peers[]` so operators can see which OAP rejected what. |
+| `404 session_not_found` | Session id unknown to **every reachable** OAP node — never created, retention timed out, or already stopped. |
+| `503 injection_disabled` | `injectionEnabled=false`; capture is permanently off until OAP restarts with the flag on. |
+| `500 registry_misconfigured` | A recorder factory wiring bug — file an issue. |
+
+## Configuration reference
+
+See [Configuration vocabulary](../configuration-vocabulary.md) for the full
+table. The DSL-debug-relevant keys:
+
+| Key | Default | Purpose |
+|--------------------------------------|---------|---------------------------------------------------------------|
+| `SW_ADMIN_SERVER` | (empty) | Enables the shared admin HTTP host. Must be set. |
+| `SW_DSL_DEBUGGING` | (empty) | Enables the DSL debug API. |
+| `SW_DSL_DEBUGGING_INJECTION_ENABLED` | `true` | Boot-time codegen switch. Set `false` to disable probes. |
+
+> SECURITY: capture payloads include sensitive operational state (raw log
+> bodies, parsed maps, MAL builder state, OAL source events). Treat the
+> admin port as authenticated infrastructure — see
+> [Admin API readme — Security Notice](readme.md#security-notice).
diff --git a/docs/en/setup/backend/admin-api/readme.md b/docs/en/setup/backend/admin-api/readme.md
new file mode 100644
index 0000000000..0d5c10e0f1
--- /dev/null
+++ b/docs/en/setup/backend/admin-api/readme.md
@@ -0,0 +1,134 @@
+# Admin API
+
+`admin-server` is SkyWalking OAP's shared HTTP host for **admin / on-demand
+write APIs** — the operations operators perform from a jumphost or CI rather
+than telemetry-time queries. This document is the entry point for every
+endpoint hosted there.
+
+## ⚠️ Security notice
+
+> The admin port has **no built-in authentication** in this release.
+> Anything that can reach the port can call every endpoint listed below.
+> Several endpoints (DSL debug capture, runtime-rule writes) expose or
+> modify operationally-sensitive state.
+>
+> **Operators MUST:**
+>
+> 1. **Gateway-protect the admin port** with an IP allow-list and an
+> authenticating reverse proxy (sidecar, gateway, mTLS terminator).
+> SkyWalking ships `admin-server` *disabled by default* (`SW_ADMIN_SERVER`
+> selector empty); enabling it is an explicit operator decision.
+> 2. **Bind to a private interface.** Set `SW_ADMIN_SERVER_HOST` to a private
+> address (or `127.0.0.1`) — never the public-facing interface.
+> 3. **Never expose the admin port to the public internet.** No exception.
+> 4. **Audit access.** The reverse proxy in front of admin-server is the
+> canonical access log for these endpoints; OAP itself does not record
+> per-request authorization decisions.
+>
+> The DSL debug API (below) is read-side and captures telemetry payloads
+> including raw log bodies, parsed maps, extracted output fields, and
+> typed builder state. Treat the admin port as authenticated infrastructure
+> and the captured payloads as sensitive data.
+
+## Endpoints by feature
+
+`admin-server` itself only provides the HTTP host. Each feature module
+registers its own routes on it. Today there are two:
+
+### [Runtime Rule Hot-Update API](runtime-rule.md)
+
+Operators add, override, inactivate, and delete MAL / LAL rule files at
+runtime without restarting OAP. Routes live under `/runtime/rule/*`.
+Requires `SW_RECEIVER_RUNTIME_RULE=default` (which in turn requires
+`SW_ADMIN_SERVER=default`).
+
+Common operations:
+
+- `POST /runtime/rule/addOrUpdate?catalog=&name=` — push a new or updated rule.
+- `POST /runtime/rule/inactivate?catalog=&name=` — turn a rule off.
+- `POST /runtime/rule/delete?catalog=&name=` — remove a rule.
+- `GET /runtime/rule/list` — current rules + per-node state.
+- `GET /runtime/rule/dump` — tar.gz snapshot of all rules.
+
+Full reference: [Runtime Rule Hot-Update API](runtime-rule.md).
+
+### [DSL Debug API](dsl-debugging.md)
+
+Operators run sampling debug sessions that capture how MAL / LAL / OAL
+rules transform live ingest, with per-stage / per-block payloads
+available for inspection. Routes live under `/dsl-debugging/*` and
+`/runtime/oal/*` (read-only OAL rule listing for the debugger's rule
+picker). Requires `SW_DSL_DEBUGGING=default` (which in turn requires
+`SW_ADMIN_SERVER=default`).
+
+Common operations:
+
+- `POST /dsl-debugging/session` — start a debug capture session.
+- `GET /dsl-debugging/session/{id}` — poll captured records.
+- `POST /dsl-debugging/session/{id}/stop` — stop a session.
+- `GET /dsl-debugging/status` — current posture (active sessions, injection state).
+- `GET /runtime/oal/files` — list loaded `.oal` files (debug picker source).
+- `GET /runtime/oal/rules/{ruleName}` — single OAL rule detail.
+
+Design reference: [SWIP-13](../../../swip/SWIP-13.md).
+Operator reference: [DSL Debug API](dsl-debugging.md).
+
+## Enabling
+
+`admin-server` and its feature modules ship disabled by default. Enable
+the features the deployment needs:
+
+```bash
+export SW_ADMIN_SERVER=default # the shared HTTP host
+export SW_RECEIVER_RUNTIME_RULE=default # to use /runtime/rule/*
+export SW_DSL_DEBUGGING=default # to use /dsl-debugging/* and /runtime/oal/*
+```
+
+If a feature module is enabled without `admin-server`, OAP fails fast at
+startup with a `ModuleNotFoundException: admin-server` — same mechanism
+that gates every other module dependency in OAP.
+
+## Configuration
+
+`admin-server` runs on **two ports**:
+
+| Port | Default | Direction | Purpose |
+|------|---------|-----------|---------|
+| `port` | `17128` | inbound from operators | HTTP REST surface (operator API) |
+| `gRPCPort` | `17129` | peer-to-peer between OAP nodes | admin-internal gRPC bus (cluster RPCs for runtime-rule Suspend/Resume/Forward, dsl-debugging install/collect/stop) |
+
+Privileged admin RPCs ride the **dedicated `17129` gRPC server** rather than the public agent / cluster gRPC port (`core.gRPCPort`, default `11800`). This isolation is the core security property of `admin-server`: an attacker on the agent network can reach `11800` for legitimate telemetry submission, but cannot reach `17129` if the operator binds it to a private peer-to-peer interface. Admin RPCs and agent telemetry never share a blast radius.
+
+Operators MUST:
+1. Bind `port` (HTTP) only to a private operator interface; gateway-protect with an IP allow-list and authenticating reverse proxy.
+2. Bind `gRPCPort` (admin-internal) only to a private peer-to-peer network; NEVER expose to operators or to the agent network.
+3. Use the same `gRPCPort` value on every OAP node — peers are dialed at the LOCAL config's port, uniform across the cluster by convention.
+
+```yaml
+admin-server:
+ selector: ${SW_ADMIN_SERVER:-}
+ default:
+ # HTTP REST surface (operator API).
+ host: ${SW_ADMIN_SERVER_HOST:0.0.0.0}
+ port: ${SW_ADMIN_SERVER_PORT:17128}
+ contextPath: ${SW_ADMIN_SERVER_CONTEXT_PATH:/}
+ idleTimeOut: ${SW_ADMIN_SERVER_IDLE_TIMEOUT:30000}
+ acceptQueueSize: ${SW_ADMIN_SERVER_QUEUE_SIZE:0}
+ httpMaxRequestHeaderSize: ${SW_ADMIN_SERVER_HTTP_MAX_REQUEST_HEADER_SIZE:8192}
+ # Admin-internal gRPC bus (peer-to-peer cluster RPCs only).
+ gRPCHost: ${SW_ADMIN_SERVER_GRPC_HOST:0.0.0.0}
+ gRPCPort: ${SW_ADMIN_SERVER_GRPC_PORT:17129}
+ gRPCMaxConcurrentCallsPerConnection: ${SW_ADMIN_SERVER_GRPC_MAX_CONCURRENT_CALL:0}
+ gRPCMaxMessageSize: ${SW_ADMIN_SERVER_GRPC_MAX_MSG_SIZE:52428800}
+ gRPCThreadPoolSize: ${SW_ADMIN_SERVER_GRPC_THREAD_POOL_SIZE:0}
+ gRPCSslEnabled: ${SW_ADMIN_SERVER_GRPC_SSL_ENABLED:false}
+ gRPCSslKeyPath: ${SW_ADMIN_SERVER_GRPC_SSL_KEY_PATH:""}
+ gRPCSslCertChainPath: ${SW_ADMIN_SERVER_GRPC_SSL_CERT_CHAIN_PATH:""}
+ gRPCSslTrustedCAsPath: ${SW_ADMIN_SERVER_GRPC_SSL_TRUSTED_CAS_PATH:""}
+ # Default per-call timeout (ms) for admin-internal RPCs between OAP nodes
+ # (dsl-debugging install / collect / stop). Workflow-specific call sites
+ # — runtime-rule Suspend / Forward — keep their own tuned values.
+ internalCommunicationTimeout: ${SW_ADMIN_SERVER_INTERNAL_COMM_TIMEOUT:5000}
+```
+
+See each feature's API reference for its own configuration block.
diff --git a/docs/en/setup/backend/backend-runtime-rule-api.md b/docs/en/setup/backend/admin-api/runtime-rule.md
similarity index 95%
rename from docs/en/setup/backend/backend-runtime-rule-api.md
rename to docs/en/setup/backend/admin-api/runtime-rule.md
index d94fc87938..74bfeea4e0 100644
--- a/docs/en/setup/backend/backend-runtime-rule-api.md
+++ b/docs/en/setup/backend/admin-api/runtime-rule.md
@@ -10,29 +10,45 @@ every node in an OAP cluster within 30 s by default.
> [Runtime Rule Hot-Update — Architecture](../../concepts-and-designs/runtime-rule-hot-update.md).
> This page focuses on the REST API surface.
-## ⚠️ Security notice
+## Security notice
-The admin port has **no authentication** in this release. The module is therefore
-**disabled by default**; enabling it opens an HTTP endpoint that can change metric and
-log-processing rules while OAP is running.
-
-Operators enabling the module MUST:
-
-1. Gateway-protect the port with an IP allow-list and separate authentication rules.
-2. Never expose port 17128 to the public internet.
-3. Bind the HTTP server to `localhost` or a private network interface if remote access is
- not required.
+The unified security notice for `admin-server` and every endpoint hosted on it
+(including this one) lives in [Admin API → Introduction](readme.md#-security-notice).
+Read it before enabling this module — it covers gateway protection, private-interface
+binding, the public-internet posture, and access auditing.
## Enabling the module
-Set the selector to `default` in `application.yml` or via env var:
+Set the selectors to `default` in `application.yml` or via env vars (the runtime-rule
+routes mount on the shared `admin-server` HTTP host, so both modules must be enabled):
```bash
+SW_ADMIN_SERVER=default
SW_RECEIVER_RUNTIME_RULE=default
```
-Default port is `17128`. All config knobs are in `application.yml` under the
-`receiver-runtime-rule` block — host, port, periodic-scan interval, self-heal threshold.
+The runtime-rule module uses **two ports** owned by `admin-server`:
+- `17128` (HTTP REST) — operator-facing admin surface; what your `curl` /
+ scripts call.
+- `17129` (admin-internal gRPC) — peer-to-peer cluster RPCs only; carries
+ the Suspend / Resume / Forward fan-out between OAP nodes during a
+ STRUCTURAL apply or a non-main forward. NEVER expose to operators or
+ to the agent network.
+
+Both are configured under the `admin-server.default:` block in
+`application.yml` (`host`/`port` for HTTP, `gRPCHost`/`gRPCPort` for the
+admin gRPC bus). The `receiver-runtime-rule` block keeps the
+runtime-rule-specific knobs only — periodic-scan interval and self-heal
+threshold.
+
+> **Why a dedicated `17129` port for cluster RPCs.** Privileged admin
+> RPCs (Suspend / Resume / Forward) used to ride the public agent /
+> cluster gRPC port (`core.gRPCPort`, default `11800`). That port is
+> reachable from the agent network for legitimate telemetry submission,
+> which means an attacker on the agent network could in principle invoke
+> Suspend/Forward. `admin-server.gRPCPort` is bound to a private
+> peer-to-peer interface only, so the admin RPCs no longer share a blast
+> radius with agent telemetry.
## HTTP surface
diff --git a/docs/en/setup/backend/configuration-vocabulary.md b/docs/en/setup/backend/configuration-vocabulary.md
index 6effdea9db..dbc0218151 100644
--- a/docs/en/setup/backend/configuration-vocabulary.md
+++ b/docs/en/setup/backend/configuration-vocabulary.md
@@ -348,15 +348,28 @@ It divided into several modules, each of which has its own settings. The followi
| - | - | enableTLS | Indicate if enable HTTPS for the server [...]
| - | - | tlsKeyPath | TLS key path [...]
| - | - | tlsCertChainPath | TLS certificate chain path [...]
-| receiver-runtime-rule | default | - | Hot-update admin endpoint for MAL / LAL rule files. **Disabled by default**: leave the selector empty to keep the provider unloaded; set `SW_RECEIVER_RUNTIME_RULE=default` to enable. The endpoint has **no authentication** in this iteration — gateway-protect with IP allow-lists [...]
-| - | - | restHost | Binding IP of the runtime-rule admin endpoint. [...]
-| - | - | restPort | Binding port of the runtime-rule admin endpoint. [...]
-| - | - | restContextPath | Web context path of the runtime-rule admin endpoint. [...]
-| - | - | restIdleTimeOut | Connector idle timeout of the runtime-rule admin endpoint (in milliseconds). [...]
-| - | - | restAcceptQueueSize | ServerSocketChannel backlog of the runtime-rule admin endpoint. [...]
-| - | - | httpMaxRequestHeaderSize | Maximum length of all HTTP/1 request headers accepted by the admin endpoint (bytes). [...]
-| - | - | reconcilerIntervalSeconds | Period (seconds) of the cluster reconcile tick. Each node periodically re-reads stored rules and reconciles its local state against the DAO. [...]
+| admin-server | default | - | Shared HTTP host for admin / on-demand write APIs (runtime-rule hot-update, DSL debug session). **Disabled by default**: leave the selector empty to keep the host closed; set `SW_ADMIN_SERVER=default` to enable. The host has **no built-in authentication** — gateway-protect wit [...]
+| - | - | host | Binding IP of the admin HTTP host. [...]
+| - | - | port | Binding port of the admin HTTP host. [...]
+| - | - | contextPath | Web context path of the admin HTTP host. [...]
+| - | - | idleTimeOut | Connector idle timeout of the admin HTTP host (in milliseconds). [...]
+| - | - | acceptQueueSize | ServerSocketChannel backlog of the admin HTTP host. [...]
+| - | - | httpMaxRequestHeaderSize | Maximum length of all HTTP/1 request headers accepted by the admin host (bytes). [...]
+| - | - | gRPCHost | Binding IP of the admin-internal gRPC bus. Carries peer-to-peer cluster RPCs for runtime-rule (Suspend/Resume/Forward) and dsl-debugging (install/collect/stop). MUST be a private peer-to-peer interface — NEVER expose to operators or to the agent network. Separate from the publ [...]
+| - | - | gRPCPort | Binding port of the admin-internal gRPC bus. Every OAP node in the cluster MUST use the same value — peers are dialed at the LOCAL config's port, uniform across the cluster by convention. | SW_ADMIN_SERVER_GRPC_PORT | 17129 [...]
+| - | - | gRPCMaxConcurrentCallsPerConnection | Max concurrent RPCs per peer connection (0 = gRPC default). [...]
+| - | - | gRPCMaxMessageSize | Max single RPC message size (bytes). [...]
+| - | - | gRPCThreadPoolSize | Server-side worker pool size for the admin gRPC bus (0 = derive from CPU count). [...]
+| - | - | gRPCSslEnabled | TLS for the admin-internal gRPC bus. [...]
+| - | - | gRPCSslKeyPath | Server private key path (PEM). [...]
+| - | - | gRPCSslCertChainPath | Server certificate chain path (PEM). [...]
+| - | - | gRPCSslTrustedCAsPath | CA bundle path (PEM); used by the admin gRPC client to validate peer certs. [...]
+| - | - | internalCommunicationTimeout | Default per-call timeout (ms) for admin-internal RPCs between OAP nodes — dsl-debugging install / collect / stop fan-outs and any other admin-internal RPC that doesn't override its own deadline. Workflow-specific call sites (runtime-rule Suspend, Forward) keep their own tuned [...]
+| receiver-runtime-rule | default | - | Hot-update admin endpoint for MAL / LAL rule files. REST routes mount on the `admin-server` HTTP host (port 17128); cluster Suspend/Resume/Forward RPCs ride the `admin-server` gRPC bus (port 17129). **Disabled by default**: leave the selector empty to keep the provider unloade [...]
+| - | - | refreshRulesPeriod | Period (seconds) of the runtime-rule refresh tick on each OAP node. Every tick the node re-reads the stored rule rows from the storage DAO and reconciles its local state — catch-up for nodes that missed a Suspend/Resume/Forward broadcast (network glitch, restart, late-joining [...]
| - | - | selfHealThresholdSeconds | Time (seconds) a SUSPENDED rule waits before the self-heal backstop forces it back to RUNNING. [...]
+| dsl-debugging | default | - | DSL debug API (sampling debugger for MAL / LAL / OAL rules). REST routes mount on the `admin-server` HTTP host (port 17128) — read-only OAL listing under `/runtime/oal/*`, session control plane under `/dsl-debugging/*`; cluster install/collect/stop RPCs ride the `admin-server` [...]
+| - | - | injectionEnabled | Boot-time switch for capture-probe emission in the OAL / MAL / LAL code generators. Default `true` — once the operator has enabled the `dsl-debugging` module (which is itself disabled by default via `SW_DSL_DEBUGGING`), probes fire and capture sessions record samples. Set `fal [...]
| ai-pipeline | default | | [...]
| - | - | uriRecognitionServerAddr | The address of the URI recognition server. [...]
| - | - | uriRecognitionServerPort | The port of the URI recognition server. [...]
diff --git a/docs/en/swip/SWIP-13.md b/docs/en/swip/SWIP-13.md
new file mode 100644
index 0000000000..db34c4a92f
--- /dev/null
+++ b/docs/en/swip/SWIP-13.md
@@ -0,0 +1,2950 @@
+# SWIP-13 Live Debugger for MAL / LAL / OAL
+
+> **Status:** implemented in 10.5.0. Operator reference:
+> [DSL Debug API](../setup/backend/admin-api/dsl-debugging.md).
+
+## Motivation
+
+SkyWalking has three DSLs that turn ingested telemetry into L1 outputs (metrics,
+logs, records ready for downstream aggregation/persistence): **MAL** (meter analyzer),
+**LAL** (log analyzer), and **OAL** (observability analysis). All three are compiled at
+OAP boot or via runtime-rule hot-update; the operator authoring the rule then has to
+infer correctness from downstream effects — "did the metric appear in the dashboard?",
+"did the log get classified into the right layer?", "why is `endpoint_cpm` reading zero
+on this endpoint?".
+
+This SWIP is scoped to **data-parsing debugging** — we capture how each DSL transforms
+its input into the L1 output that the rule emits. We do **not** follow the data past
+L1: there is no L2 aggregation, no cross-node merge, no storage round-trip in scope.
+The boundary is the `store` / emit stage at the tail of each DSL's pipeline. End-to-end
+debugging (L1 → L2 → query → UI) is out of scope.
+
+When a rule misbehaves the operator's only tools today are:
+
+- read the YAML / `.oal` source,
+- guess which clause is wrong,
+- enable DEBUG logs and grep the OAP log for hints,
+- adjust + redeploy + wait.
+
+This is acceptable for static rules edited once a release. With **runtime-rule
+hot-update** (PR #13851) shipping per-file edits in seconds, the iteration loop
+becomes the main cost. We need a debugger that shows what the live ingest actually
+does at every clause of the rule.
+
+This SWIP proposes a **passive sampling debugger** that captures the actual data
+flowing through MAL / LAL / OAL pipelines on demand — from the raw input the DSL
+receives, through every clause, to the L1 output the rule emits — returns a
+step-by-step visualization payload to the UI, and costs effectively nothing when not
+in use.
+
+## Architecture Graph
+
+```
+ ┌─────────── operator UI / swctl / curl ───────────┐
+ │ POST /dsl-debugging/session { clientId, ... } │
+ │ GET /dsl-debugging/session/{id} → payload │
+ └──────────────────────────┬───────────────────────┘
+ │ admin port (default 17128)
+ ┌────────────────────────────────────────┼──────────────────────────────────┐
+ │ │ │
+ ▼ ▼ ▼
+ ┌────────────────────────┐ bus ┌────────────────────────┐ bus ┌────────────────────────┐
+ │ OAP node A (entry) │ ─────────► │ OAP node B (peer) │ ─────────► │ OAP node C (peer) │
+ │ │ │ │ │ │
+ │ ┌──────────────────┐ │ │ ┌──────────────────┐ │ │ ┌──────────────────┐ │
+ │ │ admin-server │ │ │ │ admin-server │ │ │ │ admin-server │ │
+ │ │ (Armeria HTTP) │ │ │ │ (Armeria HTTP) │ │ │ │ (Armeria HTTP) │ │
+ │ │ /runtime/rule/* │ │ │ │ /runtime/rule/* │ │ │ │ /runtime/rule/* │ │
+ │ │ /dsl-debugging/*│ │ │ │ /dsl-debugging/*│ │ │ │ /dsl-debugging/*│ │
+ │ └──┬─────────┬─────┘ │ │ └──┬─────────┬─────┘ │ │ └──┬─────────┬─────┘ │
+ │ │ │ │ │ │ │ │ │ │ │ │
+ │ ▼ ▼ │ │ ▼ ▼ │ │ ▼ ▼ │
+ │ runtime- dsl- │ │ runtime- dsl- │ │ runtime- dsl- │
+ │ rule debugging │ │ rule debugging │ │ rule debugging │
+ │ plugin plugin │ │ plugin plugin │ │ plugin plugin │
+ │ │ │ │ │ │ │ │ │
+ │ ▼ │ │ ▼ │ │ ▼ │
+ │ GateHolder per rule │ │ GateHolder per rule │ │ GateHolder per rule │
+ │ ─ gate (volatile) │ │ ─ gate (volatile) │ │ ─ gate (volatile) │
+ │ ─ recorders[] (CoW) │ │ ─ recorders[] (CoW) │ │ ─ recorders[] (CoW) │
+ │ │ │ │ │ │ │ │ │
+ │ ┌───────────▼──────┐ │ │ ┌───────────▼──────┐ │ │ ┌───────────▼──────┐ │
+ │ │ generated MAL / │ │ │ │ generated MAL / │ │ │ │ generated MAL / │ │
+ │ │ LAL / OAL byte- │ │ │ │ LAL / OAL byte- │ │ │ │ LAL / OAL byte- │ │
+ │ │ code calls │ │ │ │ code calls │ │ │ │ code calls │ │
+ │ │ {MAL,LAL,OAL}- │ │ │ │ {MAL,LAL,OAL}- │ │ │ │ {MAL,LAL,OAL}- │ │
+ │ │ Debug.capture* │ │ │ │ Debug.capture* │ │ │ │ Debug.capture* │ │
+ │ └───────────┬──────┘ │ │ └───────────┬──────┘ │ │ └───────────┬──────┘ │
+ │ │ │ │ │ │ │ │ │
+ │ ▼ │ │ ▼ │ │ ▼ │
+ │ per-session │ │ per-session │ │ per-session │
+ │ JSON record list │ │ JSON record list │ │ JSON record list │
+ └────────────────────────┘ └────────────────────────┘ └────────────────────────┘
+ ▲
+ │ collect partials over admin-internal gRPC bus on GET
+ │
+ operator's GET request (UI / curl)
+```
+
+A new `oap-server/server-admin/` parent groups the admin-side modules.
+Three sibling modules under it cooperate, plus `server-core` provides
+the classpath contract:
+
+- **`server-admin/admin-server`** — a new shared module providing one
+ Armeria HTTP server, one admin port (default `17128`), and a
+ route-registration SPI. Hosts every *admin* REST surface
+ (write/dangerous APIs that operators run from a jumphost or CI).
+ Replaces the runtime-rule plugin's own embedded HTTP server.
+- **`server-admin/runtime-rule`** — the existing runtime-rule plugin,
+ *moved* from `server-receiver-plugin/skywalking-runtime-rule-receiver-plugin/`
+ into `server-admin/` as a sibling of `admin-server`. URL paths and YAML
+ block name stay the same; only the Maven module location changes.
+ Registers its REST handlers on `admin-server`.
+- **`server-core`** (existing module, additive only) — gains the
+ cross-DSL primitives shared by every generated rule class:
+ `GateHolder`, the marker `DebugRecorder` interface,
+ and the **OAL-specific** `OALDebug` static
+ probes + `OALDebugRecorder` interface (OAL's runtime types — `ISource`,
+ `Metrics` — are already in server-core, so its probe surface can live
+ here). MAL- and LAL-specific probes can NOT live here because their
+ signatures take types owned by the analyzer modules
+ (`SampleFamily`, `Sample`, `ExecutionContext`); putting them in
+ server-core would reverse the existing analyzer→server-core dependency
+ direction. They live in their own DSL's analyzer module instead.
+- **`analyzer/meter-analyzer`** (existing module, additive only) — gains
+ the `MALDebug` static probe class and the `MALDebugRecorder`
+ interface, both typed to `SampleFamily`, `Sample`, and `MeterEntity`.
+ MAL-generated bytecode references `MALDebug` directly; the analyzer
+ module is already on the classpath wherever MAL classes are loaded.
+- **`analyzer/log-analyzer`** (existing module, additive only) — gains
+ the `LALDebug` static probe class and the `LALDebugRecorder`
+ interface, both typed to `ExecutionContext` and `LALOutputBuilder`.
+ LAL-generated bytecode references `LALDebug` directly.
+- **`server-admin/dsl-debugging`** — a new feature module that provides
+ the runtime *implementation* behind the per-DSL recorder interfaces.
+ It depends on `server-core`, `analyzer/meter-analyzer`, and
+ `analyzer/log-analyzer` (this is the natural feature → analyzer →
+ core direction). Holds: `MALDebugRecorderImpl` /
+ `LALDebugRecorderImpl` / `OALDebugRecorderImpl` (one per DSL, each
+ with hand-written JSON serializers for that DSL's artifacts), the
+ per-node session registry (`Map<SessionId, DebugSession>` +
+ `Map<ClientId, SessionId>`), the install / uninstall paths that
+ mutate the holders' recorder arrays, the admin-internal gRPC fan-out
+ RPCs, and the REST handler. Registers its routes on the
+ `admin-server` HTTP host and its cluster gRPC service on the
+ admin-internal gRPC bus (both owned by `admin-server`).
+
+Capture lives **inside** the DSL-generated bytecode (one extra
+gate-guarded static call per stage boundary). The cross-DSL primitives
+(`GateHolder`, `RuleKey`, the `DebugRecorder` marker interface) live
+in `server-core`. Each DSL's probe class lives in that DSL's home
+module: `MALDebug` in `analyzer/meter-analyzer`, `LALDebug` in
+`analyzer/log-analyzer`, `OALDebug` in `server-core` (OAL runtime
+types are already there). Recorder implementations and the session
+registry live in `dsl-debugging`. Generated MAL bytecode references
+`MALDebug` from its own module, generated LAL bytecode references
+`LALDebug` from its own module, and generated OAL bytecode references
+`OALDebug` from `server-core` — every `invokestatic` resolves through
+the existing analyzer → server-core dependency direction; no reverse
+deps are introduced.
+
+Cluster fan-out rides the **admin-internal gRPC bus** (the same
+transport runtime-rule registers its `RuntimeRuleClusterServiceImpl`
+on); `dsl-debugging` ships its **own gRPC service implementation**
+(`DSLDebuggingClusterServiceImpl`) for its four RPCs
+(`InstallDebugSession` / `CollectDebugSamples` / `StopDebugSession` /
+`StopByClientId`).
+Runtime-rule's `RuntimeRuleClusterServiceImpl` stays exactly where it is
+— the two services are independent, register independently, and either
+module can be enabled or disabled without affecting the other. Both ride
+the **admin-internal gRPC bus** owned by `admin-server` (default port
+`17129`) — a dedicated transport separate from the public agent /
+cluster gRPC port (`core.gRPCPort`, default `11800`) so privileged admin
+RPCs stay out of the agent network's blast radius. Peer discovery is
+handled by `AdminClusterChannelManager` (admin-server module service),
+which dials each peer's admin-internal port via the cluster module's
+existing peer registry.
+
+The `status-query-plugin`'s existing `/debugging/*` routes (config dump,
+MQE step-through, trace step-through) stay where they are on the query
+port — those are read-only query-time inspection helpers and have a
+different audience from admin write APIs.
+
+## Proposed Changes
+
+### 1. The data contract returned to the UI
+
+The Claude Design handoff (`Runtime Rule Admin` canvas) pinned three shapes the
+UI expects. The backend is the source of truth — UI is a strict visualizer.
+
+The capture target is shaped *per DSL*, not generic — each DSL has its own
+workflow with its own terminal artifact handed to L1, and the data contract
+mirrors those terminal artifacts. **L1 / L2 / cross-node aggregation /
+persistence are explicitly out of scope.** The debugger always stops *before*
+the L1 hand-off — at the point where the rule has produced its final L1-bound
+object but hasn't yet entered the streaming/aggregation kernel.
+
+#### Kept-only capture for MAL (the filter is high-cardinality)
+
+**MAL** is the only DSL where rejected executions are dropped from
+`records[]`. MAL filters discriminate on tag values (`service_name`,
+`kind`, `component`) and in a multi-tenant or multi-component flow a
+rule routinely rejects 99% of the traffic routed to it by metric
+name. Publishing every rejected execution would burn `recordCap`
+(default 32) on noise within seconds and never reach a row that
+demonstrates the rule's actual processing. So for MAL the contract
+is: **every record represents one `SampleFamily` that passed the
+rule's filter and walked through to `meterEmit`**. Implementation:
+`MALDebugRecorderImpl.appendFilter(kept=false)` calls
+`discardCurrentExecution()`; the in-flight execution is silently
+dropped.
+
+**OAL** captures both kept and rejected executions. OAL filters are
+deterministic discriminators (CLIENT-vs-SERVER, layer matchers,
+status / latency predicates) — not high-cardinality tag noise.
+Rejected-source samples (`continueOn=false`) are the filter doing its
+job in plain view, useful for verifying that the partition logic
+fires the way the operator expects. Implementation: each
+`appendFilter` call adds a sample regardless of `kept`; on
+`kept=false` the recorder also calls `publishCurrentExecution()`
+because the FTL's `return` ends the per-source pipeline at that
+point, so the next source's `appendSource` mustn't bleed into this
+record.
+
+**LAL** has no analogue (`abort()` isn't a filter — it's a
+per-statement short-circuit); aborted logs publish their accumulated
+samples with `continueOn=false` on the abort point so operators can
+see where the rule gave up.
+
+#### MAL — two-half pipeline: `SampleFamily` transforms → `MeterSystem` build
+
+MAL is the only DSL whose terminal artifact is **not** a `SampleFamily`.
+`expression.run(samples)` returns a final SampleFamily, but the rule's L1-bound
+output is what `Analyzer.doAnalysis` builds **after** that — one
+`AcceptableValue<Long | DataTable | BucketedValues | PercentileArgument>` per
+`MeterEntity`, populated via `meterSystem.buildMetrics(...).accept(...)`. That
+`AcceptableValue` is what `meterSystem.doStreamingCalculation(...)` consumes;
+that call is L1 entry, so capture stops one line before it.
+
+### Wire shape
+
+The collect response is a hierarchy of `nodes[].records[].samples[]`. One
+record equals one full execution of the rule's pipeline (one
+`SampleFamily` for MAL, one `ISource` for OAL, one log for LAL); samples
+within a record are the probe firings in execution order.
+
+**Per-record fields** (rule-pipeline execution):
+
+| Field | Meaning |
+|----------------|--------------------------------------------------------------------------------------------------------|
+| `startedAtMs` | Wall-clock millis when the record opened (the first probe of the execution). |
+| `dsl` | Verbatim per-rule DSL text. Stamped per record so a session that spans a hot-update can render the right rule version per record. |
+| `rule` | Structured rule envelope: MAL carries `{metricPrefix, name, filter, exp, expSuffix}`; LAL/OAL carry `{ruleName, sourceLine}`. Lets operators read the rule's intent without re-resolving the source file. |
+| `samples[]` | Probe-emitted samples in execution order. |
+
+**Per-sample fields**:
+
+| Field | Meaning |
+|---------------|--------------------------------------------------------------------------------------------------|
+| `type` | `input` / `filter` / `function` / `aggregation` / `output` — the probe class. |
+| `sourceText` | The verbatim ANTLR slice of the DSL fragment that produced this sample (chain ops, filter clauses, aggregation calls, LAL statements). Lets operators grep against the source byte-for-byte. |
+| `continueOn` | `true` when the pipeline continued past this step; `false` on rejected filter branches (OAL) or `abort()` paths (LAL). MAL is kept-only — no `false` rows. |
+| `payload` | DSL-specific structured snapshot — see per-DSL contract below. |
+| `sourceLine` | 1-based line number in the source `.yaml` / `.oal` / LAL DSL block. Omitted when `0` (e.g. label-only probes — see "Line-number granularity in MAL" below). |
+
+**Per-DSL probe → sample-type mapping**, in pipeline order:
+
+| Catalog | Probe (call site) | `type` | Payload |
+|---------|----------------------------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------------------|
+| MAL | `captureFilter` (file-level) | `filter` | `SampleFamily.toJson()` after the file-level closure (kept-only: rejected branches are dropped). |
+| MAL | `captureInput` | `input` | head `SampleFamily` of the `exp:` body. |
+| MAL | `captureStage` × N | `function` | `SampleFamily` after each chain op (`sum`, `tagEqual`, `service`, …); `sourceText` is the verbatim segment without the leading dot. |
+| MAL | `captureDownsample` | `function` | `SampleFamily` after downsampling; `sourceText` is the function call (e.g. `rate("PT1M")`). |
+| MAL | `captureMeterEmit` (terminal) | `output` | terminal meter sample with metric name + entity + value + timeBucket. |
+| OAL | `captureSource` | `input` | `ISource.toJson()` — full source columns under `fields{}` (e.g. `ServiceRelation` source attributes). |
+| OAL | `captureFilter` | `filter` | post-filter source view; **both** kept (`continueOn=true`) **and** rejected (`continueOn=false`) branches are emitted (filters are deterministic discriminators). |
+| OAL | `captureAggregation` | `aggregation` | `Metrics.toJson()` after the aggregation function (`cpm()`, `percentile2(10)`, …). |
+| OAL | `captureEmit` (terminal) | `output` | `Metrics.toJson()` with `count` / `total` / `value` / `timeBucket` from the metric family's `appendDebugFields` hook. |
+| LAL | `appendText` (entry) | `input` | raw input only (`LogData` rendered via `LogDataDebugDump`); fires before `LALOutputBuilder.init()`. |
+| LAL | `appendParser` / `appendExtractor` | `function` | block-mode synopses — `LALOutputBuilder.outputToJson()` snapshot of the builder so far. |
+| LAL | `appendLine` | `function` | statement-mode probe (one per extractor statement); `sourceText` carries the verbatim statement; payload is the builder snapshot. |
+| LAL | `appendOutputRecord` (terminal) | `output` | final builder snapshot — DB-bound row with merged `tags[]` (each carries `status: original | lal-added | lal-override`). |
+| LAL | `appendOutputMetric` (terminal) | `output` | the SampleFamily handed off to MAL. |
+
+**LAL input/output payload split.** The first sample on every LAL record
+carries only the raw input; all subsequent samples carry only the
+output (`LALOutputBuilder.outputToJson()`). The framework renders raw
+input directly off `ctx.input()` on the input probe — the
+`LALOutputBuilder` is not consulted there because `init()` runs at the
+sink stage. This keeps the wire small (no redundant input on every
+probe) and avoids the stale-cache empty-input bug that would arise if
+the dispatch went through a not-yet-initialised builder.
+
+The terminal `output` capture is the boundary; the debugger does not
+follow the persisted entity further into L1. For MAL meter emits, the
+typed `T` (one of `Long` / `DataTable` / `BucketedValues` /
+`PercentileArgument`) is exposed in the payload so the UI can render
+the correct shape per metric type.
+
+**Downsampling — explicit vs. default.** A rule's `downsampling:` field is
+optional; when the operator omits it MAL fills in a default based on the
+metric type (typically `AVG` for gauges, `SUM` for counters; see the MAL
+compiler's metric-type resolution rules). The UI must distinguish the two
+cases so operators don't read a defaulted `AVG` as a value they declared.
+The `MalDownsample` capture carries an `origin` field — `"explicit"` when
+the YAML's `downsampling:` is present, `"default"` when MAL filled it in
+— and the UI renders the row with a "default" badge in the latter case
+(e.g., `downsampling: AVG (default)`).
+
+The UI renders the per-rule debug payload as a table whose left column is
+`sourceText` (with `sourceLine` as a small line-number gutter) and right
+column is the stage's result snapshot. Operators read the rule top-to-bottom
+in the source they wrote, with the output of each line beside it — no
+translating from synthetic stage ids back to source.
+
+**Line-number granularity in MAL.** The MAL `exp:` field in YAML is one
+scalar string — often line-folded across two or three physical lines. ANTLR
+parses the whole expression as a single source unit, so every chain stage
+emitted from that expression carries the **same `sourceLine`** (the YAML
+line where the `exp:` key sits). This is honest rather than synthetic;
+attempting to back-compute per-stage line numbers from sub-positions inside
+a folded scalar would be brittle and misleading. Per-stage discrimination
+comes from `sourceText` — the verbatim fragment is unique per stage even
+when lines collide. The UI line-gutter shows the same number twice when
+that's the truth, and operators recognise the fragments anyway because
+they wrote them.
+
+LAL is different — its DSL has one statement per line in the YAML body
+(`json {}`, `service parsed.x as String`, `tag 'k': v`, etc.), so each
+captured LAL statement has a distinct `sourceLine`. OAL captures **per
+clause** with each clause's actual ANTLR-derived line; same-line clauses
+share a number, distinct-line clauses each get their own. `sourceText`
+carries the fragment-level detail regardless. (See the OAL data
+contract below for the full per-clause table.)
+
+**File-level filter fires before `expression.run()`.** The probe ordering
+within one ingest pass is:
+
+```
+MALDebug.captureFilter (file `filter:` block — once, on the whole input map)
+MALDebug.captureInput (per metric reference inside expression.run — already filtered)
+MALDebug.captureStage × N (each chain stage)
+MALDebug.captureScope (after the expSuffix scope-binding stage)
+MALDebug.captureDownsample (rule-level metadata, fires once)
+MALDebug.captureMeterBuild (per MeterEntity, in Analyzer post-chain)
+MALDebug.captureMeterEmit (per MeterEntity, immediately before doStreamingCalculation)
+```
+
+`MALDebug.captureInput` therefore reports the **post-filter** sample count, not
+the raw count the receiver delivered — the file-level filter has already
+trimmed the input map by the time the generated `run()` reads it.
+
+#### LAL — typed output, not "unified log"
+
+LAL's terminal artifact is **the typed output builder** the rule produces.
+The codegen calls `ctx.setOutput(new <OutputType>())` at the start of
+`execute()`; the extractor populates fields via typed setters; then
+`RecordSinkListener.parse()` calls `builder.init(metadata, input,
+moduleManager)` to fill in standard fields, and `build()` calls
+`builder.complete(sourceReceiver)`. The L1 boundary is `complete(...)`;
+capture is the populated builder *just before* that call.
+
+`OutputType` is per-rule and varies — `LogBuilder`, `EnvoyAccessLogBuilder`,
+`DatabaseSlowStatementBuilder`, `SampledTraceBuilder`, custom subclasses
+contributed by `LALSourceTypeProvider` SPI. The UI must show the actual type
+name and its full populated state, not flatten everything to "LogData".
+
+A LAL rule may also emit metrics into MAL via `MetricExtractor.submitMetrics
+(ctx, sampleBuilder)`. The `Sample` produced is wrapped into a `SampleFamily`
+and either appended to `ctx.metricsContainer()` (when called inside a MAL
+fanout) or fed to `provider.getMetricConverts().forEach(it -> it.toMeter(...))`.
+This second emission is independent of the typed record output — some rules
+emit both, some only one, some neither.
+
+Per-record blocks (operator toggles per-block sampling; off blocks return
+`null` cells):
+
+| Block | Captured artifact |
+|-----------------|-------------------------------------------------------------------------------------------------------------------|
+| `text` | raw log body string + body-type (`json` / `yaml` / `text` / `none`) |
+| `parser` | parsed map / typed proto object — the structure `parsed.*` resolves against |
+| `extractor` | populated typed output builder (snapshot mid-flight) + extracted tags + resolved `def` variables |
+| `sink` | branch taken (`enforcer` / `sampler` / `rateLimit`) + `kept` / `dropped` + reason |
+| `output_record` | the **typed** output builder right before `complete()` — includes `outputType` class name + every populated field |
+| `output_metric` | the `SampleFamily` (≥ 1 `Sample`) about to be handed to MAL — only present when the rule has a `metrics{}` block |
+
+`output_record` and `output_metric` are independent — each is `null` if the
+rule didn't produce that output (`output_record` is `null` on a sink-dropped
+record; `output_metric` is `null` for any rule without `metrics{}`).
+`output_metric` does **not** chain into a separate MAL debug session.
+
+Per-record envelope:
+
+```
+{ id, ts, svc, body_type, text, parsed, extracted, def_vars,
+ sink: { branch, kept, note },
+ output_record: { outputType, builderState } | null,
+ output_metric: { sampleFamily } | null }
+```
+
+##### Granularity — block vs. statement (opt-in)
+
+The block view above (5 fixed blocks + 2 outputs) answers most operator
+questions: did the parser succeed → did the extractor populate fields →
+did the sink keep the record. For "step through every DSL line in my
+extractor" investigations, sessions can opt into **statement-level**
+capture via a request-time flag.
+
+| Granularity | Capture points per log record | Record-cap math |
+|----------------------|----------------------------------------------------------------------------------------------------------------|--------------------------------------------------|
+| `block` (default) | 5 fixed blocks + 2 output emit points = 7 cells per log record | 8 records × 7 = 56 cells worst case |
+| `statement` (opt-in) | 7 block points **plus** one cell per meaningful DSL statement (each `service`/`tag`/`def`/`if`/`metrics` line) | 8 records × (7 + N statements) cells worst case |
+
+`statement` mode burns the per-session record cap faster — a rule with 12
+extractor statements + sink branches produces ≈ 19 cells per log record
+versus 7 in block mode. The
+[per-session record cap](#5-session-lifecycle-storage-and-memory-control)
+treats each cell uniformly, so an operator selecting `statement` should
+lower `recordCap` in the request body to stay within the session's
+worst-case heap budget. The session payload reports
+`granularity: "block" | "statement"` so the UI knows which renderer to
+use.
+
+Statement-level capture maps each cell to its source DSL line via
+`sourceLine` carried through `LALScriptModel`'s AST nodes (see
+[LAL hooks](#34-generator-hook-insertion--concrete-code-diffs)) — the UI
+fetches the rule YAML once at session start (via `/runtime/rule/list`)
+and reads line `sourceLine` directly. **No `sourceText` is duplicated in
+the captured payload.** LAL's one-statement-per-line convention guarantees
+the line is unique per captured statement, so a line number alone is
+enough — unlike MAL chain stages and OAL clauses, which often share a
+single line and therefore must carry their verbatim fragment as
+`sourceText`.
+
+This granularity flag applies to **LAL only**. MAL is already
+statement-level for free — every method-chain stage is one DSL statement
+and the chain capture (`MALDebug.captureStage`) records it. OAL is single-rule
+already (`do<Metric>(source)` is one logical statement), so the
+distinction does not apply.
+
+#### OAL — Source row → typed `Metrics` object before `in()`
+
+OAL is dispatcher-style: each `do<MetricsName>(source)` method takes one
+`Source` row, optionally filters it, builds a `<MetricsName>Metrics` object,
+copies fields from the source, calls the entry method (`combine` /
+`accept` / `add` depending on the function class), and finally calls
+`MetricsStreamProcessor.getInstance().in(metrics)`. That `in(...)` call is
+the L1 boundary; capture stops one line before it.
+
+Operator selects **one OAL line**
+(e.g. `endpoint_cpm = from(Endpoint.*).cpm();`) and configures
+`(maxRows, windowSec)`. The picker is fed by the
+[OAL read-only management API](#41-oal-read-only-management-api--the-catalog-the-debugger-picks-from)
+introduced in this SWIP — without it the UI has no way to discover what
+OAL rules exist at runtime.
+
+OAL probe → sample-type mapping (one source event = one record;
+samples flow source → filter → aggregation → output in execution order):
+
+| Probe (call site) | `type` | Origin | Captured artifact |
+|--------------------------|---------------|--------------------|-------------------------------------------------------------------------------------------------------------------|
+| `captureSource` | `input` | OAL clause | `ISource.toJson()` — the inbound source row (concrete subtype: `Endpoint`, `Service`, `ServiceRelation`, …) under `payload.fields{}`. `sourceText` = `from(...)` clause. |
+| `captureFilter` | `filter` | OAL clause (× N) | post-filter source view; both kept (`continueOn=true`) and rejected (`continueOn=false`) branches are emitted. One sample per filter clause in the rule. |
+| `captureAggregation` | `aggregation` | OAL clause | `Metrics.toJson()` after the aggregation function. `sourceText` = the verbatim clause (`cpm()`, `percentile2(10)`, `apdex(name, status)`). |
+| `captureEmit` (terminal) | `output` | compiler-implicit | `Metrics.toJson()` of the L1-ready metric (`count` / `total` / `value` / `timeBucket` from `Metrics.appendDebugFields`); fired immediately before `MetricsStreamProcessor.in(metrics)`. |
+
+**Per-clause is the natural OAL granularity, not an opt-in.** OAL is one
+logical rule per line in the `.oal` file, but each rule is composed of
+distinct clauses (`from(Source.*)`, optionally `.filter(...)`, ending in
+`.<aggregationFn>(...)`). Every user-written clause becomes its own captured
+row carrying its verbatim OAL fragment as `sourceText` — same principle as
+MAL chain stages, just naturally bounded (≤ 5 stages per rule). There is no
+`granularity: "block" | "statement"` flag for OAL because the only useful
+view is per-clause; there's nothing finer to opt into.
+
+Some rules write the whole expression on one physical line
+(`endpoint_cpm = from(Endpoint.*).cpm();`); others split it across lines
+for readability. Per-clause `sourceLine` reports each clause's actual
+position from ANTLR's source location — operators see distinct lines when
+they wrote them on distinct lines, and the same line repeated when they
+wrote the rule on one line. `sourceText` discriminates clauses regardless.
+
+The `output` sample is the terminal capture (compiler-emitted, no
+`sourceText` — render with an `(implicit)` badge in the UI). The
+aggregation function (`CPMFunction`, `LongAvgFunction`,
+`PercentileFunction`, etc.) determines what fields the `Metrics`
+instance carries at emit time. `Metrics.appendDebugFields(JsonObject)`
+is the per-family hook (`CPMMetrics` emits `count` / `total` / `value`,
+`PercentileMetrics` emits its bucket array, etc.); the UI sees the
+concrete class name plus the populated state. The debugger does **not**
+follow the metric into the cross-node L2 / persistence path; cross-node
+aggregation is out of scope.
+
+For the per-DSL examples and full payload key reference see the
+operator pages:
+[MAL](../setup/backend/admin-api/dsl-debugging-mal.md),
+[LAL](../setup/backend/admin-api/dsl-debugging-lal.md),
+[OAL](../setup/backend/admin-api/dsl-debugging-oal.md).
+
+### 2. Capture mechanism — per-rule-instance `GateHolder` (no ThreadLocal, no shared registry)
+
+Captures live entirely in the rule's runtime state, not in a per-thread
+context. **Each compiled rule class generates its own `GateHolder`
+instance at construction time**, with `contentHash` set as a constructor
+argument and made immutable for that holder's lifetime. Probes embedded
+in the generated bytecode read the holder directly:
+
+```java
+public final class GateHolder {
+ // Hot-path field: read by every probe call site. Public + volatile so
+ // the generated bytecode reads it directly with one volatile load and
+ // the JIT can hoist the load when the method is hot.
+ public volatile boolean gate;
+
+ // Immutable for this holder's lifetime — set by constructor at codegen,
+ // never overwritten. Hot-update creates a brand-new holder with the
+ // new content's hash; the old holder retains its own hash until GC.
+ private final String contentHash;
+
+ // CoW snapshot of currently-bound recorders; mutated only on session
+ // install / uninstall via the synchronized methods below. Public +
+ // volatile so generated probe code reads it with one volatile load —
+ // symmetric with `gate`. Readers MUST treat the array as immutable;
+ // writers always replace the reference, never mutate the array in place.
+ public volatile DebugRecorder[] recorders = EMPTY;
+ private static final DebugRecorder[] EMPTY = new DebugRecorder[0];
+
+ public GateHolder(String contentHash) { this.contentHash = contentHash; }
+
+ public String getContentHash() { return contentHash; }
+
+ /** Called by `DebugSessionRegistry.install(...)`. */
+ public synchronized void addRecorder(DebugRecorder r) {
+ DebugRecorder[] old = recorders;
+ DebugRecorder[] neu = Arrays.copyOf(old, old.length + 1);
+ neu[old.length] = r;
+ recorders = neu;
+ if (old.length == 0) gate = true; // 0→1 transition flips the gate
+ }
+
+ /** Called by `DebugSessionRegistry.uninstall(...)`. */
+ public synchronized void removeRecorder(DebugRecorder r) {
+ DebugRecorder[] old = recorders;
+ DebugRecorder[] neu = removeFromArray(old, r);
+ recorders = neu;
+ if (neu.length == 0) gate = false; // 1→0 transition flips the gate
+ }
+}
+```
+
+`contentHash` is `final` because the **holder belongs to one specific
+generated class instance**. There is no scenario in which the same
+holder serves two different rule contents: hot-update produces a new
+class, the new class's constructor produces a new holder with the new
+hash, and the old holder continues to exist (with the old hash) for as
+long as the old class instance is referenced by in-flight ingest. Each
+captured record stamps `record.contentHash = holder.getContentHash()`
+at append time and inherently picks up the right value because the
+probe fires on the holder owned by whichever class the receiver thread
+is running.
+
+```java
+// idiom emitted by codegen at every probe call site:
+if (DEBUG.gate) { // public volatile field — direct read
+ MALDebug.captureStage(DEBUG, rule, sourceLine, sourceText, family);
+}
+```
+
+`DEBUG` is **not `static final`** for MAL and LAL — those DSLs each
+have a singleton runtime instance per rule (`MalExpression`,
+`LalExpression`), so the holder lives as a regular instance field on
+that singleton:
+
+```java
+// inside a generated MAL rule class:
+public final GateHolder DEBUG = new GateHolder("7c3a91…");
+
+// inside a generated LAL rule class:
+public final GateHolder DEBUG = new GateHolder("9b12a0…");
+```
+
+For OAL, the per-metric runtime artifact (`<MetricName>Metrics`) is
+constructed *per source row* — not a singleton. Instead, the
+dispatcher singleton (e.g., `EndpointDispatcher`) holds **per-metric
+holder instance fields**, one per metric the dispatcher routes to:
+
+```java
+// inside a generated OAL dispatcher (singleton, held by SourceReceiverImpl):
+public final GateHolder gate_endpoint_cpm = new GateHolder("8a21f0…");
+public final GateHolder gate_endpoint_resp_time = new GateHolder("8a21f0…");
+// ... one per metric in this scope's `do<Metric>` method set
+
+private void do_endpoint_cpm(Endpoint source) {
+ if (gate_endpoint_cpm.gate) {
+ OALDebug.captureSource(gate_endpoint_cpm, "endpoint_cpm", ...);
+ }
+ // ... existing build / aggregation / emit ...
+}
+```
+
+(All metrics in one OAL file share the file's `contentHash`, so the
+constructor argument is the same for every per-metric holder created
+from that file.)
+
+The install / uninstall paths call `addRecorder(...)` /
+`removeRecorder(...)` directly — no reflection. The methods are
+`synchronized` to serialise the array copy-on-write against concurrent
+mutations; the receiver hot-path read takes no lock (it just reads the
+volatile array reference). The gate is auto-managed inside
+`addRecorder` / `removeRecorder` on 0↔1 transitions, so the install
+path doesn't have to remember to flip it separately.
+
+**No shared `Map<RuleKey, GateHolder>` registry.** Sessions find the
+live holder by asking the rule's *existing* analyzer / dispatcher
+infrastructure for the live runtime artifact (the `MalExpression` /
+`LalExpression` / dispatcher singleton) and then calling
+`.debugHolder(...)` on it — see
+[Install / uninstall](#33-install--uninstall--direct-gateholder-mutation-no-per-thread-context).
+This is what makes content-hash-overwrite-on-hot-update structurally
+impossible: there is no shared map entry to overwrite, no swap window,
+no race. Old captures continue on the old holder until the old class
+goes out of scope (typically a few hundred ms during drain); new
+captures fire on the new holder. Records carry whichever
+`contentHash` was on the holder they were stamped from — the UI sees a
+clean version boundary in the session payload.
+
+When no session is bound to the rule, `gate=false` and JIT eliminates the
+call site entirely — ~1 ns idle cost (the volatile load + branch). When
+sessions are bound, the probe iterates the (small) recorders array and
+each recorder appends to its own session's payload. **Multiple sessions
+on the same rule are first-class:** each is one entry in the array, all
+fed by the same probe call.
+
+The receiver path is unchanged. **There is no ThreadLocal**, no
+analyzer-side wrapper, no per-thread context to install or propagate.
+Sessions register themselves on the rule's holder; receiver threads
+running the rule's generated code read the holder directly. This sits
+inside the project's "no ThreadLocal side-channels" rule documented in
+`CLAUDE.md` — the holder is rule-scoped state, not a thread-local
+side-channel.
+
+### 3. Capture API and per-DSL hooks
+
+This section nails down (a) the static surface generated code calls, (b) the
+recorder interface that owns per-session state, (c) the receiver-side scope
+wrapper that installs / uninstalls a recorder for one ingest pass, and (d)
+exactly what the MAL / LAL / OAL generators emit.
+
+#### 3.1 Per-DSL probe classes — `MALDebug`, `LALDebug`, `OALDebug`
+
+The probe surface splits into three DSL-specific classes, each living
+in the same module that owns the DSL's runtime types. There is no
+single unified `DSLDebug` class — generated bytecode for each DSL
+calls into its own DSL's probe class. This is forced by Java's
+classpath direction: `SampleFamily` and `Sample` are owned by
+`analyzer/meter-analyzer`; `ExecutionContext` is owned by
+`analyzer/log-analyzer`; both modules depend on `server-core`, not the
+other way around.
+
+| Probe class | Module | Generated bytecode origin | Imports analyzer types |
+|----------------|----------------------------|------------------------------|------------------------|
+| `MALDebug` | `analyzer/meter-analyzer` | MAL classes (compiled by this module) | `SampleFamily`, `Sample` (own module) |
+| `LALDebug` | `analyzer/log-analyzer` | LAL classes (compiled by this module) | `ExecutionContext` (own module) |
+| `OALDebug` | `server-core` | OAL classes (compiled by `oal-rt`, deps server-core) | none — `ISource`, `Metrics` are in server-core |
+
+Each probe takes the per-rule `GateHolder` (server-core) as its first
+argument and fans the capture out to every recorder registered on the
+holder. **No ThreadLocal**, no per-thread context propagation. The
+generated bytecode wraps each call site with the gate check
+(`if (DEBUG.gate)`) so the call is only emitted once a session is active
+for this rule.
+
+The probe surface is intentionally **not** generic. Each DSL has its own
+workflow with its own typed L1-bound artifact, and we want generated code
+to call methods whose types match the artifact rather than boxing
+everything through a `StageData` wrapper.
+
+##### `MALDebug` (in `analyzer/meter-analyzer`, package `org.apache.skywalking.oap.meter.analyzer.v2.dsldebug`)
+
+```java
+public final class MALDebug {
+ private MALDebug() {}
+
+ // ── Part A: chain probes (emitted from MALMethodChainCodegen / MALExprCodegen)
+ public static void captureInput(GateHolder h, String rule, int sourceLine,
+ String metricRef, SampleFamily f) { fanOut(h, r -> ((MALDebugRecorder) r).appendInput(rule, sourceLine, metricRef, f)); }
+ public static void captureFilter(GateHolder h, String rule, int sourceLine,
+ String filterExpr, SampleFamily f) { fanOut(h, r -> ((MALDebugRecorder) r).appendFilter(rule, sourceLine, filterExpr, f)); }
+ public static void captureStage(GateHolder h, String rule, int sourceLine,
+ String sourceText, SampleFamily f) { fanOut(h, r -> ((MALDebugRecorder) r).appendStage(rule, sourceLine, sourceText, f)); }
+ public static void captureScope(GateHolder h, String rule, int sourceLine, String scopeExpr,
+ SampleFamily f, Map<MeterEntity,Sample[]> meterSamples) { fanOut(h, r -> ((MALDebugRecorder) r).appendScope(rule, sourceLine, scopeExpr, f, meterSamples)); }
+ public static void captureDownsample(GateHolder h, String rule, String function,
+ String origin, SampleFamily f) { fanOut(h, r -> ((MALDebugRecorder) r).appendDownsample(rule, function, origin, f)); }
+
+ // ── Part B: post-chain probes (emitted from Analyzer.doAnalysis, hand-written)
+ public static void captureMeterBuild(GateHolder h, String rule, MeterEntity entity,
+ String metricName, String valueType,
+ AcceptableValue<?> v) { fanOut(h, r -> ((MALDebugRecorder) r).appendMeterBuild(rule, entity, metricName, valueType, v)); }
+ public static void captureMeterEmit(GateHolder h, String rule, MeterEntity entity,
+ String metricName, AcceptableValue<?> v,
+ long timeBucket) { fanOut(h, r -> ((MALDebugRecorder) r).appendMeterEmit(rule, entity, metricName, v, timeBucket)); }
+
+ private static void fanOut(GateHolder h, java.util.function.Consumer<DebugRecorder> body) {
+ DebugRecorder[] rs = h.recorders;
+ for (int i = 0; i < rs.length; i++) {
+ DebugRecorder r = rs[i];
+ if (!r.isCaptured()) body.accept(r);
+ }
+ }
+}
+```
+
+##### `LALDebug` (in `analyzer/log-analyzer`, package `org.apache.skywalking.oap.log.analyzer.v2.dsldebug`)
+
+```java
+public final class LALDebug {
+ private LALDebug() {}
+
+ public static void captureText(GateHolder h, String rule, ExecutionContext ctx) { fanOut(h, r -> ((LALDebugRecorder) r).appendText(rule, ctx)); }
+ public static void captureParser(GateHolder h, String rule, ExecutionContext ctx) { fanOut(h, r -> ((LALDebugRecorder) r).appendParser(rule, ctx)); }
+ public static void captureExtractor(GateHolder h, String rule, ExecutionContext ctx) { fanOut(h, r -> ((LALDebugRecorder) r).appendExtractor(rule, ctx)); }
+ public static void captureSink(GateHolder h, String rule, ExecutionContext ctx,
+ String branch, boolean kept) { fanOut(h, r -> ((LALDebugRecorder) r).appendSink(rule, ctx, branch, kept)); }
+ public static void captureLine(GateHolder h, String rule, int sourceLine,
+ ExecutionContext ctx) { fanOut(h, r -> ((LALDebugRecorder) r).appendLine(rule, sourceLine, ctx)); }
+ public static void captureOutputRecord(GateHolder h, String rule, ExecutionContext ctx,
+ LALOutputBuilder builder) { fanOut(h, r -> ((LALDebugRecorder) r).appendOutputRecord(rule, ctx, builder)); }
+ public static void captureOutputMetric(GateHolder h, String rule, ExecutionContext ctx,
+ SampleFamily handedToMal) { fanOut(h, r -> ((LALDebugRecorder) r).appendOutputMetric(rule, ctx, handedToMal)); }
+
+ private static void fanOut(GateHolder h, java.util.function.Consumer<DebugRecorder> body) {
+ DebugRecorder[] rs = h.recorders;
+ for (int i = 0; i < rs.length; i++) {
+ DebugRecorder r = rs[i];
+ if (!r.isCaptured()) body.accept(r);
+ }
+ }
+}
+```
+
+`captureOutputMetric` takes `SampleFamily` directly:
+`analyzer/log-analyzer` already depends on `analyzer/meter-analyzer`
+(see `oap-server/analyzer/log-analyzer/pom.xml`), so this is the
+existing dep direction, not a new one. The whole probe API stays
+typed end-to-end with no `Object` leaks.
+
+##### `OALDebug` (in `server-core`, package `org.apache.skywalking.oap.server.core.dsldebug`)
+
+OAL's runtime types (`ISource`, `Metrics`) are already owned by
+`server-core`, so OAL's probe class can live there too:
+
+```java
+public final class OALDebug {
+ private OALDebug() {}
+
+ public static void captureSource(GateHolder h, String rule, int sourceLine,
+ String fromClauseText, ISource src) { fanOut(h, r -> ((OALDebugRecorder) r).appendSource(rule, sourceLine, fromClauseText, src)); }
+ public static void captureFilter(GateHolder h, String rule, int sourceLine,
+ String filterClauseText, ISource src,
+ String left, String op, String right, boolean kept) { fanOut(h, r -> ((OALDebugRecorder) r).appendFilter(rule, sourceLine, filterClauseText, src, left, op, right, kept)); }
+ public static void captureBuild(GateHolder h, String rule, Metrics built) { fanOut(h, r -> ((OALDebugRecorder) r).appendBuild(rule, built)); }
+ public static void captureAggregation(GateHolder h, String rule, int sourceLine,
+ String aggregationClauseText, Metrics afterEntryFn) { fanOut(h, r -> ((OALDebugRecorder) r).appendAggregation(rule, sourceLine, aggregationClauseText, afterEntryFn)); }
+ public static void captureEmit(GateHolder h, String rule, Metrics readyForL1) { fanOut(h, r -> ((OALDebugRecorder) r).appendEmit(rule, readyForL1)); }
+
+ private static void fanOut(GateHolder h, java.util.function.Consumer<DebugRecorder> body) {
+ DebugRecorder[] rs = h.recorders;
+ for (int i = 0; i < rs.length; i++) {
+ DebugRecorder r = rs[i];
+ if (!r.isCaptured()) body.accept(r);
+ }
+ }
+}
+```
+
+These methods are the **only** symbols generated bytecode references —
+generators never see `DebugRecorder` or the ThreadLocal. That keeps the
+generated class identical across builds whether or not a session is active.
+
+The probe set is intentionally **closed**. New DSL stages introduced later
+(say, OAL adds a new dispatcher stage) get a new typed probe; we do not
+push it through a generic `capture(stageId, Object)` because that loses
+type information at the recorder and forces every `appendXxx` to deal
+with `instanceof` switching. Typed probes keep the recorder narrow and
+the JSON renderer per-artifact compact.
+
+#### 3.2 `DebugRecorder` — marker interface + per-DSL extensions
+
+`DebugRecorder` is split the same way as the probe classes: a marker
+interface in `server-core` (so `GateHolder.recorders[]` can be typed
+without dragging in analyzer modules), plus three extension interfaces
+in their respective DSL modules carrying the typed `appendXxx` methods.
+
+##### `DebugRecorder` marker (in `server-core`)
+
+```java
+package org.apache.skywalking.oap.server.core.dsldebug;
+
+public interface DebugRecorder {
+ String sessionId();
+ String catalog();
+ String name();
+ String ruleName();
+
+ /** Narrow gate — called by `DebugSessionRegistry` before bind. */
+ boolean matches(String catalog, String name, String ruleName);
+
+ /** Becomes true once recordCap is hit; further appends are no-ops. */
+ boolean isCaptured();
+}
+```
+
+This is the only type `GateHolder.recorders[]` is parameterised on. It
+contains no `append*` methods — those live on the per-DSL extensions
+below. Probes downcast to the right extension interface based on which
+DSL's class is calling.
+
+##### `MALDebugRecorder` (in `analyzer/meter-analyzer`, extends `DebugRecorder`)
+
+```java
+package org.apache.skywalking.oap.meter.analyzer.v2.dsldebug;
+
+public interface MALDebugRecorder extends DebugRecorder {
+ void appendInput(String rule, int sourceLine, String metricRef, SampleFamily f);
+ void appendFilter(String rule, int sourceLine, String filterExpr, SampleFamily f);
+ void appendStage(String rule, int sourceLine, String sourceText, SampleFamily f);
+ void appendScope(String rule, int sourceLine, String scopeExpr,
+ SampleFamily f, Map<MeterEntity, Sample[]> meterSamples);
+ void appendDownsample(String rule, String function, String origin, SampleFamily f);
+ void appendMeterBuild(String rule, MeterEntity entity, String metricName,
+ String valueType, AcceptableValue<?> v);
+ void appendMeterEmit(String rule, MeterEntity entity, String metricName,
+ AcceptableValue<?> v, long timeBucket);
+}
+```
+
+##### `LALDebugRecorder` (in `analyzer/log-analyzer`, extends `DebugRecorder`)
+
+```java
+package org.apache.skywalking.oap.log.analyzer.v2.dsldebug;
+
+public interface LALDebugRecorder extends DebugRecorder {
+ void appendText(String rule, ExecutionContext ctx);
+ void appendParser(String rule, ExecutionContext ctx);
+ void appendExtractor(String rule, ExecutionContext ctx);
+ void appendSink(String rule, ExecutionContext ctx, String branch, boolean kept);
+ void appendLine(String rule, int sourceLine, ExecutionContext ctx);
+ void appendOutputRecord(String rule, ExecutionContext ctx, LALOutputBuilder builder);
+ void appendOutputMetric(String rule, ExecutionContext ctx, SampleFamily handedToMal);
+}
+```
+
+##### `OALDebugRecorder` (in `server-core`, extends `DebugRecorder`)
+
+```java
+package org.apache.skywalking.oap.server.core.dsldebug;
+
+public interface OALDebugRecorder extends DebugRecorder {
+ void appendSource(String rule, int sourceLine, String fromClauseText, ISource src);
+ void appendFilter(String rule, int sourceLine, String filterClauseText,
+ ISource src, String left, String op, String right, boolean kept);
+ void appendBuild(String rule, Metrics built);
+ void appendAggregation(String rule, int sourceLine, String aggregationClauseText,
+ Metrics afterEntryFn);
+ void appendEmit(String rule, Metrics readyForL1);
+}
+```
+
+##### Implementation: one impl class per DSL in `dsl-debugging`
+
+Each session targets one rule which is one DSL. The session's recorder
+implements only the corresponding extension interface:
+
+- `MALDebugRecorderImpl implements MALDebugRecorder` — for sessions on `otel-rules` / `log-mal-rules`.
+- `LALDebugRecorderImpl implements LALDebugRecorder` — for sessions on `lal`.
+- `OALDebugRecorderImpl implements OALDebugRecorder` — for sessions on `oal`.
+
+`server-admin/dsl-debugging` depends on `server-core`,
+`analyzer/meter-analyzer`, and `analyzer/log-analyzer`, so it can
+construct any of the three impls. The `instanceof` downcast inside the
+probe is a compile-time-trivial check on a tagged type — JIT specialises
+it after warm-up.
+
+`DebugSession` (in `dsl-debugging`) is the recorder factory plus the
+`Map<SessionId, DebugSession>` and `Map<ClientId, SessionId>`
+book-keeping from
+[Session lifecycle, storage, and memory control](#5-session-lifecycle-storage-and-memory-control).
+Serialization happens at capture time inside each `appendXxx`
+implementation — payload JSON is built once per probe so polls during
+the session return identical bytes every time.
+
+#### 3.3 Install / uninstall — direct `GateHolder` mutation, no per-thread context
+
+There is no analyzer-side wrapper. Sessions are installed and removed
+by mutating the per-rule `GateHolder` from the session-management code
+in `dsl-debugging`; receiver threads observe the new state via the
+volatile `recorders` field on the next probe. The receiver code path
+itself is unchanged from today.
+
+**Rule identity is a typed object, not a slash-encoded string.**
+A small immutable class lives in `server-core` and is the single
+identifier the install/uninstall path uses — every method takes
+`RuleKey`, never an encoded `String`:
+
+```java
+public record RuleKey(Catalog catalog, String name, String ruleName) {
+ public enum Catalog { OTEL_RULES, LOG_MAL_RULES, LAL, OAL }
+}
+```
+
+`RuleKey.equals/hashCode` are auto-derived from the three fields; no
+delimiter, no escaping, no parsing. Each DSL's runtime artifact
+exposes its own `RuleKey ruleKey()` method (filled in at codegen
+time), so the install path can inspect the live artifact's identity
+without re-parsing strings.
+
+**Each `DebugSession` stores the exact `GateHolder` it bound into.**
+That lets uninstall remove the recorder from the *original* holder
+even if the rule has since been hot-updated and the live artifact now
+references a different holder. Without this, install would bind to V1
+and uninstall would call `removeRecorder` on V2 (the new live holder),
+leaving V1's recorder array and gate stuck on until the old class is
+garbage-collected.
+
+```java
+public final class DebugSession {
+ final SessionId sessionId;
+ final ClientId clientId;
+ final RuleKey ruleKey;
+ final GateHolder boundHolder; // ← captured at install time, never updated
+ // ... payload, lifecycle state ...
+}
+
+// inside DebugSessionRegistry.install(key, clientId, sessionSpec):
+GateHolder h = liveHolderFor(key); // current live holder (see lookup below)
+DebugRecorder recorder = newRecorderFor(key, sessionSpec);
+DebugSession session = new DebugSession(sessionId, clientId, key, h, ...);
+sessions.put(sessionId, session);
+clientIdIndex.put(clientId, sessionId);
+h.addRecorder(recorder); // synchronized; auto-flips gate on 0→1
+
+// inside DebugSessionRegistry.uninstall(sessionId):
+DebugSession session = sessions.remove(sessionId);
+clientIdIndex.remove(session.clientId, sessionId);
+session.boundHolder.removeRecorder(session.recorder); // ← original holder
+```
+
+Hot-update interactions stay correct:
+
+- Install runs against the live holder of the moment. `session.boundHolder` is
+ pinned to that instance.
+- A subsequent runtime-rule swap creates a new live holder; the registry's
+ next `liveHolderFor(key)` returns the new one. New session installs bind
+ to the new holder.
+- The old session's `uninstall` (manual stop / retention / replaced-by-client)
+ walks through `session.boundHolder` — i.e., the V1 holder — so V1's
+ recorder array shrinks correctly and V1's gate flips back to `false` when
+ its last recorder leaves. The new live holder is untouched.
+- The old V1 class's drainage continues hitting V1's holder; once V1's
+ recorder array is empty its gate is `false` and probes are JIT-eliminated
+ again on V1 too.
+
+The `liveHolderFor(key)` lookup goes through each DSL's existing
+runtime registry — no new shared `Map<RuleKey, GateHolder>`:
+
+```java
+private GateHolder liveHolderFor(RuleKey key) {
+ switch (key.catalog()) {
+ case OTEL_RULES:
+ case LOG_MAL_RULES:
+ // The meter-analyzer Provider already maintains the live
+ // MalExpression instance per rule, replaced on hot-update.
+ return malProvider.liveExpression(key).debugHolder();
+
+ case LAL:
+ // The log-analyzer Provider already maintains live LalExpression
+ // instances per rule, replaced on hot-update.
+ return lalProvider.liveExpression(key).debugHolder();
+
+ case OAL:
+ // The dispatcher is held by SourceReceiverImpl; per-metric holder
+ // lives on it as `gate_<ruleName>` instance field.
+ return dispatcherManager.dispatcherFor(key)
+ .debugHolder(key.ruleName());
+ }
+ throw new IllegalArgumentException(key.toString());
+}
+```
+
+No reflection. `addRecorder` / `removeRecorder` are typed methods on
+the `GateHolder` instance reachable through each DSL's existing live-
+artifact registry — direct method calls. Hot-update for MAL or LAL
+replaces the live `MalExpression` / `LalExpression` instance with a
+new one carrying its own fresh `GateHolder`; pre-update sessions stay
+on the old holder (which the old instance still references) until
+their session window ends or the old class drains; post-update
+sessions install on the new holder. There is no shared registry to
+overwrite mid-flight.
+
+`recorders` is a copy-on-write snapshot — readers iterate whatever
+snapshot was visible when they read the volatile reference; writers
+publish a new snapshot atomically. The per-rule lock serialises
+concurrent install/uninstall against each other (rare events; one
+operator-driven action at a time per rule). The receiver hot path takes
+no lock and incurs no TL access.
+
+Why no analyzer-side wrapper:
+
+- The receiver doesn't need to know which sessions match the rule —
+ the holder *is* the per-rule binding. Probes consult it directly.
+- No indirection to add at every analyzer call site (no diff per
+ analyzer beyond the codegen change inside the rule's own class).
+- Multi-session support is built into the recorder array: install
+ appends, uninstall removes; the probe iterates whatever's there. No
+ composite-recorder type, no per-thread state to install or
+ propagate.
+
+#### 3.4 Generator hook insertion — concrete code diffs
+
+##### MAL — two halves: chain codegen + Analyzer hand-off to MeterSystem
+
+The MAL pipeline splits cleanly across two files. The chain (filter / closure /
+scope / tag / downsampling stages) is emitted by `MALMethodChainCodegen`; the
+post-chain MeterSystem build + emit lives in `Analyzer.doAnalysis`. The
+debugger hooks both halves.
+
+**Chain codegen** ([source][mal-chain]). The current loop emits one
+`_var = _var.method(args);` line per stage. The hook is one extra line per
+stage:
+
+```java
+// before:
+for (final MethodCall mc : chain) {
+ if (mc.isExtension()) { emitExtensionCall(sb, var, mc); continue; }
+ sb.append(" ").append(var).append(" = ")
+ .append(var).append('.').append(mc.getName()).append('(');
+ emitBuiltinArgs(sb, var, mc);
+ sb.append(");\n");
+}
+
+// after:
+for (final MethodCall mc : chain) {
+ if (mc.isExtension()) {
+ emitExtensionCall(sb, var, mc);
+ emitMalCapture(sb, var, mc, /*op=*/mc.getNamespace() + "::" + mc.getName());
+ continue;
+ }
+ sb.append(" ").append(var).append(" = ")
+ .append(var).append('.').append(mc.getName()).append('(');
+ emitBuiltinArgs(sb, var, mc);
+ sb.append(");\n");
+ emitMalCapture(sb, var, mc, mc.getName());
+}
+
+private static void emitMalCapture(StringBuilder sb, String var, MethodCall mc) {
+ // Emit a literal-string call. `mc.toMalSource()` rebuilds the verbatim
+ // MAL fragment from the AST: e.g. "tagEqual('region', 'us-east-1')",
+ // ".sum(['svc', 'instance'])", "myext::scale(2.0)". The compiled string
+ // literal goes into the constant pool once per generated class.
+ sb.append(" ").append(DSL_DEBUG_FQN)
+ .append(".MALDebug.captureStage(\"").append(currentRuleName).append("\", ")
+ .append(mc.getLine()).append(", \"")
+ .append(escapeJava(mc.toMalSource())).append("\", ")
+ .append(var).append(");\n");
+}
+```
+
+`MALDebug.captureInput` is emitted once at the top of `run()` (in
+`MALExprCodegen.generate*`, where `_var` first reads from `samples.get(...)`)
+with `sourceLine` = the line where the metric reference appeared and
+`metricRef` = the reference text (e.g. `node_memory_MemAvailable_bytes`).
+
+Filter (file-level, applied in `Analyzer` *before* `expression.run()`), tag,
+binary operations (`* 100`, `/ otherMetric`, etc., emitted by
+`MALExprCodegen` as their own `_var = _var.multiply(...)` / `_var.divide(...)`
+lines), and built-in chain stages each get a `MALDebug.captureStage(...)` line
+carrying their MAL fragment. Scope binding emits `MALDebug.captureScope(...)` with
+the `expSuffix` text (e.g. `instance(['host'], Layer.VM)`) plus the freshly
+populated `Map<MeterEntity, Sample[]>`. Closure companion classes
+(`MALClosureCodegen`) need no changes — the chain stage that owns the
+closure (`tag({...})`, `forEach({...})`) already captures it at its method
+call line; the closure's own line shows up as the `sourceText` of that
+parent stage.
+
+**File-level filter, also in `Analyzer.doAnalysis`** ([source][mal-analyzer]).
+The `filter:` block at the top of the YAML file (`{ tags -> tags.job ==
+'node-exporter' }`) is applied to the input map *before* `expression.run()`
+fires. One probe captures the filtered SampleFamily with the filter's verbatim
+text:
+
+```java
+// existing:
+if (filterExpression != null) {
+ input = filterExpression.filter(input);
+ if (input.isEmpty()) return;
+}
+
+// after — every probe takes the rule's GateHolder as its first argument:
+GateHolder DEBUG = expression.debugHolder(); // resolved once per ingest pass
+if (filterExpression != null) {
+ input = filterExpression.filter(input);
+ if (DEBUG.gate) {
+ MALDebug.captureFilter(DEBUG, ruleName, fileFilterLine,
+ fileFilterSource /* "{ tags -> tags.job == 'node-exporter' }" */,
+ SampleFamily.flatten(input) /* lightweight view across the input map */);
+ }
+ if (input.isEmpty()) return;
+}
+```
+
+`fileFilterLine` and `fileFilterSource` are stored on the rule definition
+when the YAML is parsed; they're per-rule constants, captured once per
+ingest pass.
+
+**Post-chain hand-off in `Analyzer.doAnalysis`**. The existing code iterates
+`meterSamples` and calls `meterSystem.buildMetrics(...).accept(...)` followed
+by `send(...)` (which calls `meterSystem.doStreamingCalculation(...)`). Two
+probes wrap the build / emit boundary:
+
+```java
+// existing:
+meterSamples.forEach((meterEntity, ss) -> {
+ generateTraffic(meterEntity);
+ switch (metricType) {
+ case single:
+ AcceptableValue<Long> sv = meterSystem.buildMetrics(metricName, Long.class);
+ sv.accept(meterEntity, getValue(ss[0]));
+ send(sv, ss[0].getTimestamp()); // ← doStreamingCalculation = L1 entry
+ break;
+ // ... labeled / histogram / histogramPercentile branches
+ }
+});
+
+// after — every probe takes the rule's GateHolder as its first argument:
+final GateHolder DEBUG = expression.debugHolder();
+meterSamples.forEach((meterEntity, ss) -> {
+ generateTraffic(meterEntity);
+ switch (metricType) {
+ case single:
+ AcceptableValue<Long> sv = meterSystem.buildMetrics(metricName, Long.class);
+ sv.accept(meterEntity, getValue(ss[0]));
+ if (DEBUG.gate) {
+ MALDebug.captureMeterBuild(DEBUG, ruleName, meterEntity, metricName, "Long", sv);
+ MALDebug.captureMeterEmit(DEBUG, ruleName, meterEntity, metricName, sv,
+ TimeBucket.getMinuteTimeBucket(ss[0].getTimestamp()));
+ }
+ send(sv, ss[0].getTimestamp());
+ break;
+ // ... same shape for labeled (DataTable), histogram (BucketedValues),
+ // histogramPercentile (PercentileArgument)
+ }
+});
+```
+
+Both probes fire **before** `send(...)`. Note that `send(...)` itself
+mutates `v` by setting `v.timeBucket` immediately before
+`doStreamingCalculation`, so at the moment `MALDebug.captureMeterEmit` runs
+`v.timeBucket` is still 0. The probe takes the time-bucket as a separate
+parameter, computed from the same `ss[0].getTimestamp()` that `send` will
+use; the recorder serializes `(v fields, computed timeBucket)` so the UI
+sees the exact `(value, timeBucket)` pair that's about to be streamed,
+without needing the post-mutation snapshot. The debugger does not follow
+the value past `doStreamingCalculation`.
+
+##### LAL — `LALClassGenerator.generateExecuteMethod` + Analyzer-side terminal
+
+LAL's `execute(filterSpec, ctx)` already passes `ctx` explicitly — no
+ThreadLocal needed. The codegen places per-block probes inside `execute()`
+([source][lal-exec]); the **terminal** typed-output capture lives in
+`RecordSinkListener.parse` so it fires once `init(metadata, ...)` has
+populated standard fields and right before the listener's `build()`
+delegates to `complete(sourceReceiver)`.
+
+```java
+// generated execute() body — hooks are gate-guarded and pass `this.DEBUG`:
+public final GateHolder DEBUG = new GateHolder("...sha...");
+
+public void execute(FilterSpec filterSpec, ExecutionContext ctx) {
+ LalRuntimeHelper h = new LalRuntimeHelper(ctx);
+ h.ctx().setOutput(new <OutputType>());
+ if (DEBUG.gate) LALDebug.captureText(DEBUG, ruleName, ctx); // >>> raw body
+ filterSpec.json(ctx);
+ if (DEBUG.gate) LALDebug.captureParser(DEBUG, ruleName, ctx); // >>> parsed map
+ if (!ctx.shouldAbort()) {
+ _extractor(filterSpec.extractor(), h);
+ if (DEBUG.gate) LALDebug.captureExtractor(DEBUG, ruleName, ctx); // >>> typed builder mid-flight
+ }
+ filterSpec.sink(ctx);
+ if (DEBUG.gate) LALDebug.captureSink(DEBUG, ruleName, ctx, // >>> kept / dropped + branch
+ h.lastSinkBranch(),
+ !h.lastSinkDropped());
+}
+```
+
+`h.lastSinkBranch()` / `h.lastSinkDropped()` are tiny additions to
+`LalRuntimeHelper` that capture which sink branch (`enforcer` / `sampler` /
+`rateLimit` / pass-through) ran last and whether it dropped the record;
+they're populated by the existing sink spec methods on the same hot path.
+
+**Terminal record output** is captured one frame up the stack in
+`RecordSinkListener` ([source][lal-sink]) — i.e., not in generated bytecode
+but in the analyzer wrapper that drives it:
+
+```java
+// before:
+public LogSinkListener parse(LogMetadata md, Object input, ExecutionContext ctx) {
+ if (ctx == null || !(ctx.output() instanceof LALOutputBuilder)) return this;
+ builder = ctx.outputAsBuilder();
+ builder.init(md, input, moduleManager);
+ return this;
+}
+
+// after:
+public LogSinkListener parse(LogMetadata md, Object input, ExecutionContext ctx) {
+ if (ctx == null || !(ctx.output() instanceof LALOutputBuilder)) return this;
+ builder = ctx.outputAsBuilder();
+ builder.init(md, input, moduleManager);
+ GateHolder DEBUG = ctx.lalExpression().debugHolder(); // resolved per call
+ if (DEBUG.gate) {
+ LALDebug.captureOutputRecord(DEBUG, ruleName, ctx, builder); // >>> typed builder, before complete()
+ }
+ return this;
+}
+```
+
+That captures the **typed** builder (`LogBuilder`, `EnvoyAccessLogBuilder`,
+`DatabaseSlowStatementBuilder`, `SampledTraceBuilder`, custom subclasses) with
+all standard fields populated. The recorder serializes the class name plus
+every populated bean property — the UI shows the operator the actual record
+shape that's about to be persisted.
+
+**Metric output** (only present for rules with a `metrics{}` block) fires from
+`MetricExtractor.submitMetrics` ([source][lal-extractor]) right after
+`SampleFamilyBuilder` produces the family that's about to be handed to MAL:
+
+```java
+// inside submitMetrics(...):
+final Sample sample = builder.build();
+final SampleFamily sampleFamily = SampleFamilyBuilder.newBuilder(sample).build();
+GateHolder DEBUG = ctx.lalExpression().debugHolder();
+if (DEBUG.gate) {
+ LALDebug.captureOutputMetric(DEBUG, ruleName, ctx, sampleFamily); // >>> handed to MAL
+}
+// ... existing dispatch into MetricConverts ...
+```
+
+The capture fires once whether the family is appended to `ctx.metricsContainer()`
+(MAL fanout case) or fed straight to `provider.getMetricConverts().forEach(...)`.
+The debugger does **not** chain into the MAL pipeline that consumes this
+family — that boundary is the L1-bound hand-off from LAL's perspective, even
+though MAL would itself become L1-bound only after its own `meterSystem`
+build/emit. From this LAL session's view, MAL is downstream and out of scope.
+
+The two helpers `LALBlockCodegen` and `LALDefCodegen` are unchanged for
+block-level capture — block content emission is the same; only the
+orchestration in `generateExecuteMethod` adds the capture lines.
+
+**Statement-level capture — always emitted, recorder-side filter.**
+LAL codegen **unconditionally** emits one
+`LALDebug.captureLine(rule, sourceLine, ctx)` call after every
+meaningful AST statement (`OutputFieldAssignment`, `TagAssignment`,
+`DefStatement`, `IfBlock` predicate evaluations, `MetricsInline`
+blocks). The probes are present in every compiled LAL class regardless
+of how the operator will later debug the rule. **There is no
+recompile-on-session-install path** — the rule's class is generated
+once at boot or runtime-rule apply time, with all probe call sites in
+place, and stays that class until a structural runtime-rule update
+replaces it.
+
+The session's `granularity` field is purely a **recorder-side filter**:
+
+| Granularity | Recorder behaviour |
+|------------------|-------------------------------------------------------------------------------------------------------------|
+| `block` (default)| `appendLalLine(...)` returns immediately — statement records are dropped. Only block-level appends accumulate. |
+| `statement` | `appendLalLine(...)` serializes and appends the record. Block-level appends accumulate alongside. |
+
+Idle-path cost remains zero — `DEBUG.gate=false` eliminates *all* probe
+call sites, statement-level included. Active-path cost in block mode is
+slightly higher than the previous design (~7 ns per statement probe to
+do the holder load + recorder iteration + recorder's
+"granularity != statement → return" check), but the absolute cost is
+bounded: a typical LAL extractor block has 10-15 statements, so
+~100-150 ns of additional cost per log record on a block-mode session.
+Well inside the active-path budget. The
+[performance ladder](#36-performance-plan--idle-path-effectively-free-active-path-bounded) updates
+to reflect this; idle and capturing rows are unchanged.
+
+**AST plumbing required (one-time, not per session).** Add a
+`int sourceLine` field to every relevant statement node in
+`LALScriptModel`. The ANTLR listener already has
+`ctx.getStart().getLine()` per parsed statement; thread it into the
+AST. The codegen reads it when emitting the `LALDebug.captureLine(...)` call.
+The bytecode `LineNumberTable` already produced by
+`LALClassGenerator.addLineNumberTable` tracks **generated-Java** lines
+and is unrelated — `sourceLine` here is the original DSL line in the
+YAML so the UI can highlight the corresponding source line side-by-side
+with the captured value.
+
+**Concurrency** — multiple sessions of any granularity mix on the same
+rule: each registers as one entry on the rule's `GateHolder.recorders`
+array (see [Performance plan](#36-performance-plan--idle-path-effectively-free-active-path-bounded)).
+Each statement probe iterates the array; recorders running in `block`
+mode drop the record; recorders running in `statement` mode keep it.
+No cross-session interference; no class swap; no DSLClassLoaderManager
+involvement.
+
+##### OAL — FreeMarker template `dispatcher/doMetrics.ftl`
+
+OAL emits one `do<Metric>(source)` method per metric per dispatcher class,
+filter as a guard, then `<Metric>Metrics metrics = new ...`, source-field
+copy, `metrics.<entryFn>(...)`, then `MetricsStreamProcessor.in(metrics)`
+([source][oal-tpl]). Hooks bracket each transition; the final probe fires
+**before** the L1-entry `in(...)` call, not after:
+
+The template uses the full per-clause probe signatures (rule, sourceLine,
+sourceText, ...) as defined in
+[the probe surface](#31-per-dsl-probe-classes--maldebug-laldebug-oaldebug).
+
+###### OAL AST deliverable — source-position fields don't exist today
+
+The current OAL model classes do not carry source positions:
+
+- `oap-server/oal-rt/src/main/java/.../v2/model/SourceReference.java` —
+ has no `location` / `sourceLine` field.
+- `oap-server/oal-rt/src/main/java/.../v2/model/FunctionCall.java` —
+ same; no source location.
+- `oap-server/oal-rt/src/main/java/.../v2/model/FilterExpression.java` —
+ declares a `SourceLocation location` field (line 42), but
+ `OALListenerV2.enterFilterStatement` (line 99) does not populate it
+ on construction; the field is currently always null.
+
+This SWIP makes the source-position carrying an **explicit deliverable**
+of Phase 3 (OAL single-node):
+
+1. Add `int sourceLine` (or `SourceLocation`) to `SourceReference` and
+ `FunctionCall` model classes (analogous to `FilterExpression`'s
+ existing field).
+2. Update `OALListenerV2.enterSourceReference` / `enterFunctionCall` /
+ `enterFilterStatement` to populate the source position from
+ `ctx.getStart().getLine()` on each AST node it constructs.
+3. Add `String sourceText()` accessors that pretty-print the verbatim
+ OAL fragment from each node (`from(Endpoint.*)`, `.cpm()`,
+ `.filter(detectPoint == DetectPoint.SERVER)`, etc.) — used by the
+ FreeMarker model for `${from.sourceText}`, `${f.sourceText}`,
+ `${aggregationSourceText}`.
+
+These are pure model-side additions; no behavioural change to the OAL
+runtime, no breaking change to the parser. The SourceReference /
+FunctionCall changes are tracked under Phase 3 in
+[Phased delivery](#10-phased-delivery).
+
+Once the AST changes land, `${from.line}`, `${aggregationLine}`,
+`${f.line}` populate from `MetricDefinition.SourceReference` /
+`FilterExpression` / `FunctionCall` nodes through the FreeMarker
+model.
+
+Each per-metric holder lives on the dispatcher singleton as
+`gate_<metricsName>` (declared once in the dispatcher's class body
+alongside the other instance fields the template emits). The
+`do<MetricsName>` body reads that holder and passes it as the first arg
+to every `OALDebug.captureXxx` call:
+
+```ftl
+private void do${metricsName}(${sourcePackage}${from.sourceName} source) {
+ if (gate_${metricsName}.gate) {
+ org.apache.skywalking.oap.server.core.dsldebug.OALDebug
+ .captureSource(gate_${metricsName}, "${metricsName}",
+ ${from.line}, "${from.sourceText}", source);
+ }
+
+<#if filters.filterExpressions??>
+ <#list filters.filterExpressions as f>
+ if (!new ${f.expressionObject}().match(${f.left}, ${f.right})) {
+ if (gate_${metricsName}.gate) {
+ org.apache.skywalking.oap.server.core.dsldebug.OALDebug
+ .captureFilter(gate_${metricsName}, "${metricsName}",
+ ${f.line}, "${f.sourceText}",
+ source, "${f.left}", "${f.opSymbol}", "${f.right}", false);
+ }
+ return;
+ }
+ if (gate_${metricsName}.gate) {
+ org.apache.skywalking.oap.server.core.dsldebug.OALDebug
+ .captureFilter(gate_${metricsName}, "${metricsName}",
+ ${f.line}, "${f.sourceText}",
+ source, "${f.left}", "${f.opSymbol}", "${f.right}", true);
+ }
+ </#list>
+</#if>
+
+ ${metricsClassPackage}${metricsName}Metrics metrics =
+ new ${metricsClassPackage}${metricsName}Metrics();
+ <#-- ... existing source-field copy ... -->
+ if (gate_${metricsName}.gate) {
+ org.apache.skywalking.oap.server.core.dsldebug.OALDebug
+ .captureBuild(gate_${metricsName}, "${metricsName}", metrics);
+ }
+
+ metrics.${entryMethod.methodName}(${entryMethod.argsList});
+ if (gate_${metricsName}.gate) {
+ org.apache.skywalking.oap.server.core.dsldebug.OALDebug
+ .captureAggregation(gate_${metricsName}, "${metricsName}",
+ ${aggregationLine}, "${aggregationSourceText}", metrics);
+ org.apache.skywalking.oap.server.core.dsldebug.OALDebug
+ .captureEmit(gate_${metricsName}, "${metricsName}", metrics);
+ }
+ org.apache.skywalking.oap.server.core.analysis.worker
+ .MetricsStreamProcessor.getInstance().in(metrics);
+}
+```
+
+`OALDebug.captureEmit` fires immediately **before** `MetricsStreamProcessor.in(metrics)`
+— that call is L1 entry, and the debugger stops there. The dispatcher template
+`dispatch.ftl` is unchanged: it just routes a `Source` row to its `do<Metric>`
+methods; capture is per-metric, inside each method.
+
+No receiver-side wrapping is needed for OAL — the dispatcher
+singleton holds per-metric `GateHolder` instance fields
+(`gate_endpoint_cpm`, `gate_endpoint_resp_time`, …), and each
+`do<Metric>(source)` reads its own metric's holder directly. Same
+self-resolving probe model as MAL and LAL; the receiver code path
+(`SourceReceiverImpl.receive` → `DispatcherManager.forward` →
+`dispatcher.dispatch(source)` → the generated `do<Metric>(source)`)
+is unchanged.
+
+[mal-chain]: ../../../oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALMethodChainCodegen.java
+[mal-analyzer]: ../../../oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/Analyzer.java
+[lal-exec]: ../../../oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALClassGenerator.java
+[lal-sink]: ../../../oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/provider/log/listener/RecordSinkListener.java
+[lal-extractor]: ../../../oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/spec/extractor/MetricExtractor.java
+[oal-tpl]: ../../../oap-server/oal-rt/src/main/resources/code-templates-v2/dispatcher/doMetrics.ftl
+
+#### 3.5 Recap of what changes vs. what doesn't
+
+| Component | Change |
+|-----------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| MAL / LAL / OAL grammar + AST | unchanged |
+| MAL / LAL / OAL generator orchestrators | one extra capture-emission helper called per stage |
+| Generated bytecode | adds N static-method calls to the per-DSL probe class (`MALDebug.capture*` / `LALDebug.capture*` / `OALDebug.capture*`) |
+| Receiver / analyzer call sites | unchanged — probes self-resolve through the per-rule `GateHolder` |
+| New plumbing | `GateHolder`, `DebugRecorder` marker, `OALDebug` + `OALDebugRecorder` (in `server-core`); `MALDebug` + `MALDebugRecorder` (in `analyzer/meter-analyzer`); `LALDebug` + `LALDebugRecorder` (in `analyzer/log-analyzer`); recorder impls + `DebugSessionRegistry` + REST handler (in `dsl-debugging`); HTTP host shared via `admin-server`. **No new shared `Map<RuleKey, GateHolder>` registry** — sessions reach holders through each DSL's existing live-artifa [...]
+| Behavior when no session is installed | per-rule `GateHolder` (singleton) gates every probe via its `gate` volatile-boolean field; JIT eliminates the call when off (~1 ns idle). See [Performance plan](#36-performance-plan--idle-path-effectively-free-active-path-bounded). |
+
+Compile-failure handling is unchanged — capture hooks only fire if the rule
+compiled successfully. The chat clarification is honored: "if it is parsed,
+there is no grammar issue".
+
+#### 3.6 Performance plan — idle path effectively free, active path bounded
+
+Capture lives in the data hot path. A receiver-side LAL pipeline can run
+100K log lines/sec; a busy OAL dispatcher fans dozens of `Source` rows per
+trace. Probe overhead must be near-zero when no session is active and
+strictly bounded when one is. Three layers handle this:
+
+##### Layer 1 — per-rule-instance `GateHolder`, gate flipped through interface methods
+
+Every generated rule artifact carries a `GateHolder` field constructed
+fresh per class generation. The holder is bound to *that specific
+class instance*, not shared across class versions. Concrete shape per
+DSL:
+
+- **MAL / LAL**: `public final GateHolder DEBUG = new GateHolder("...sha...");`
+ on the generated rule class. The class has a singleton runtime
+ instance (`MalExpression` / `LalExpression`); the holder lives as an
+ instance field on that singleton.
+- **OAL**: per-metric `public final GateHolder gate_<metric> = new GateHolder("...sha...");`
+ on the dispatcher singleton (one holder per metric the dispatcher
+ routes to). The metric class itself is per-source-row, so the holder
+ cannot live there; the dispatcher is the per-rule control point.
+
+In both cases, hot-update replacing the rule's runtime artifact also
+replaces the holder — the new artifact's constructor builds a brand-new
+`GateHolder` with the new content's hash. The old holder lingers as
+long as the old artifact is referenced by drainage and session state;
+its `contentHash` is `final` and never overwritten, so records captured
+on the old artifact carry the right (old) hash.
+
+The DSL codegen wraps every probe call site:
+
+```java
+// emitted in MALMethodChainCodegen / LALClassGenerator / OAL FreeMarker:
+if (DEBUG.gate) {
+ MALDebug.captureStage(DEBUG, ruleName, sourceLine, sourceText, _var);
+}
+```
+
+`DEBUG.gate` is a public volatile boolean field on the holder, default
+`false`. When `DEBUG.gate == false` (the steady state on every node not
+currently debugging the rule), the JIT compiles the branch as
+never-taken: the capture call site becomes effectively dead code, and
+the only runtime cost per probe is one volatile-load + branch (~1 ns).
+On a hot LAL rule with statement-level probes and no session running,
+the per-log-line idle overhead is `~15 probes × 1 ns = 15 ns` — well
+under the 1% budget.
+
+**Session install flips the gate through `GateHolder`'s typed
+interface methods — never reflection.** The session code calls
+`holder.addRecorder(r)`; the holder atomically appends the recorder to
+its CoW array and flips `gate=true` if it just transitioned 0→1:
+
+```java
+// inside DebugSessionRegistry.install(ruleKey, recorder):
+GateHolder h = liveHolderFor(key);
+h.addRecorder(recorder); // synchronized; auto-flips gate on 0→1
+```
+
+Symmetric on uninstall: `h.removeRecorder(recorder)` removes the entry
+and flips `gate=false` on 1→0. The holder owns its own state
+transitions; callers don't have to remember to flip the gate
+separately.
+
+The `gate` field is one writer-side write per 0↔1 transition, many
+reader-side loads per probe; `volatile` is the correct primitive. The
+`recorders` array is replaced via volatile write on each
+addRecorder/removeRecorder, so receiver threads always see a coherent
+snapshot.
+
+##### Layer 2 — holder-resident recorder array, fan-out at the probe
+
+When `gate` is `true`, the probe call site fires and reaches
+the relevant per-DSL probe (`MALDebug.captureXxx` / `LALDebug.captureXxx`
+/ `OALDebug.captureXxx`, depending on which DSL's class is calling).
+That method reads the holder's
+`recorders` array and iterates:
+
+```java
+DebugRecorder[] rs = h.recorders; // one volatile load
+for (int i = 0; i < rs.length; i++) {
+ DebugRecorder r = rs[i];
+ if (!r.isCaptured()) body.accept(r); // each recorder appends to its session
+}
+```
+
+No `ThreadLocal`. No per-thread state to install or maintain. Multiple
+sessions on the same rule sit in the array and each receives the
+capture independently. The receiver thread doesn't know or care which
+sessions exist — it just iterates the holder's snapshot.
+
+##### Layer 3 — per-session `isCaptured()` short-circuit
+
+Once a session hits its record cap, `isCaptured()` flips to `true`. The
+fan-out loop still iterates over that recorder, but skips the
+serialization. With a single capped session in a 1-recorder array, the
+total cost stays in the few-ns range (gate + array read + isCaptured
+check + skip).
+
+##### Cost summary
+
+| State | Cost per probe |
+|--------------------------------------------------------------|-------------------------------------------------------------------------|
+| Idle — no session anywhere on this rule (`gate=false`) | ~1 ns (volatile load + branch; JIT eliminates the call site) |
+| Active — 1 recorder, all captured (cap reached) | ~7 ns (gate + array load + isCaptured + skip) |
+| Active — N recorders, all captured | ~7 ns + ~3 ns per extra recorder |
+| Active — capturing (real append, single recorder) | ~500–2000 ns (JSON serialize + append) |
+| Active — capturing across N recorders | N × ~500–2000 ns (independent per recorder) |
+
+Active-path serialization uses per-DSL streaming renderers — direct
+`StringBuilder` walks of `SampleFamily` / `ExecutionContext` /
+`LALOutputBuilder` / `Metrics` rather than reflection-based JSON. The
+serializer is the same code path that produced the
+[characters-not-bytes ceiling](#why-characters-and-not-bytes), so memory
+and CPU are bounded by the same closed-form formula.
+
+##### Bytecode + memory overhead when not running
+
+| Resource | Cost |
+|-----------------------------|-----------------------------------------------------------------------------------|
+| Generated bytecode per rule | ~20 bytes per probe call site (gate-load + branch + invokestatic + args) |
+| Constant-pool entries | One String entry per `sourceText` literal; one int per `sourceLine` |
+| Per-class field | One static-final `GateHolder` reference (instance carries `volatile boolean gate`, `volatile String contentHash`, `volatile DebugRecorder[] recorders`) |
+| Session registry state | Empty when no sessions exist — no idle memory |
+
+Across the bundled rule set (~50 MAL rules, ~14 LAL rules, ~12 OAL files
+× many rules each), the cumulative bytecode increase is small — single-
+digit MiB. The OAP image size is dominated by other concerns; this is
+not a constraint.
+
+##### Validation gate
+
+JMH benchmarks added under `oap-server-bench`:
+
+- `DSLDebugIdleBench` — measures probe call cost with no session installed.
+ CI gate: regress build if mean cost exceeds **2× the no-probe baseline**
+ on the same JIT.
+- `DSLDebugActiveBench` — measures probe call cost during an active
+ session. CI gate: regress build if mean cost exceeds **5 μs per
+ capture** at the median.
+- Tail-latency check (`p99 < 50 μs per capture`) under contended load
+ (8 receiver threads × hot rule).
+
+Benchmarks run on every PR that touches `oap-server/server-core/.../dsldebug/`,
+the DSL generators, or the analyzer call sites. A fail blocks merge.
+
+##### Why not a single global gate
+
+A single global "any session active anywhere" flag would also eliminate
+idle cost — but it would force every receiver thread to enter the
+fan-out path (read the recorders array, iterate, even if empty) the
+moment any session on any rule exists anywhere. With per-class gates,
+only threads processing the *specifically debugged* rule pay attention;
+everyone else stays on the JIT-eliminated call-site path.
+
+##### What this means for deployment
+
+- Operators leaving DSL-debugging enabled in `application.yml` with no
+ active session pay essentially zero — confirmed by the JMH idle bench.
+- Operators starting a session on rule X impose cost only on the receiver
+ threads that processed rule X's data — and only for the duration of
+ the session capture window plus retention.
+- A misbehaving operator script that opens many sessions concurrently is
+ bounded by `maxActiveSessions: 200` (rejected with HTTP 429 above the
+ cap), and even at the cap the cost on any one rule's hot path is
+ identical to one session.
+
+#### 3.7 Implementation strategy — hard-coded probes, hand-written serializers, no reflection
+
+The capture pipeline has three layers; only the call sites are auto-
+emitted by codegen, the rest is hand-written Java in `server-core` and
+`dsl-debugging` with no reflection on the hot path.
+
+##### Layer 1 — call sites: codegen-emitted constants
+
+Probe call sites (`if (DEBUG.gate) {MAL,LAL,OAL}Debug.captureXxx(DEBUG, ...)`) are
+**hard-coded as bytecode constants** by the existing DSL codegens:
+
+| Codegen owner | Emits |
+|---------------------------------------------|-----------------------------------------------------------------------------------------|
+| `MALMethodChainCodegen.emitChainStatements` | one `MALDebug.captureStage(...)` per chain stage; `sourceText` = `mc.toMalSource()` written into the constant pool as a String literal |
+| `MALExprCodegen` (input read) | one `MALDebug.captureInput(...)` at each `samples.get(...)` site |
+| `Analyzer.doAnalysis` (hand-written, not codegen) | the `MALDebug.captureFilter` / `MALDebug.captureScope` / `MALDebug.captureDownsample` / `MALDebug.captureMeterBuild` / `MALDebug.captureMeterEmit` calls |
+| `LALClassGenerator.generateExecuteMethod` | per-block `LALDebug.captureText` / `LALDebug.captureParser` / `LALDebug.captureExtractor` / `LALDebug.captureSink` |
+| `LALBlockCodegen.generate*Statement` | `LALDebug.captureLine(...)` after each statement (always emitted; recorder filters by `granularity`) |
+| `RecordSinkListener.parse` (hand-written) | `LALDebug.captureOutputRecord(...)` after `builder.init(...)` |
+| `MetricExtractor.submitMetrics` (hand-written) | `LALDebug.captureOutputMetric(...)` after `SampleFamilyBuilder.build()` |
+| `dispatcher/doMetrics.ftl` FreeMarker | per-clause OAL captures with line numbers from `MetricDefinition.SourceReference` etc. |
+
+Each emitted call expands to 4-8 bytecode instructions plus the
+constant-pool entries for the literal arguments (rule name, source line,
+source text). No reflection, no method-handle lookup, no allocation —
+the JVM's invokestatic instruction dispatches directly to
+the matching per-DSL probe class.
+
+##### Layer 2 — `{MAL,LAL,OAL}Debug.captureXxx`: hand-written probes in their DSL's home module
+
+Each probe is a few-line static method that fans out to the holder's
+recorder array:
+
+```java
+public static void MALDebug.captureStage(GateHolder h, String rule, int sourceLine,
+ String sourceText, SampleFamily f) {
+ DebugRecorder[] rs = h.recorders; // one volatile load
+ for (int i = 0; i < rs.length; i++) {
+ DebugRecorder r = rs[i];
+ if (!r.isCaptured()) r.appendMalStage(rule, sourceLine, sourceText, f);
+ }
+}
+```
+
+No reflection. No `ThreadLocal`. No allocation on the no-active-recorder
+path (the receiver hits this method only when `h.gate == true`, which
+implies `recorders.length >= 1`; the array load + length check is
+unconditional). When all recorders are `isCaptured()`, the loop runs
+through them in a few ns.
+
+##### Layer 3 — `DebugRecorderImpl.appendXxx`: hand-written per-DSL serializers in `dsl-debugging`
+
+Each `appendXxx` method has a hand-written JSON serializer for *that
+specific artifact type*. No `ObjectMapper`, no reflection, no annotation
+processing — direct `StringBuilder` writes that know the artifact's
+shape:
+
+```java
+@Override public void appendMalStage(String rule, int line, String text, SampleFamily f) {
+ if (++recordCount > recordCap) { captured = true; return; }
+ StringBuilder sb = new StringBuilder(256);
+ sb.append("{\"kind\":\"stage\",\"line\":").append(line)
+ .append(",\"text\":").append(jsonEscape(text))
+ .append(",\"samples\":[");
+ Sample[] samples = f.samples;
+ for (int i = 0; i < samples.length; i++) {
+ if (i > 0) sb.append(',');
+ appendSampleJson(sb, samples[i]); // hand-written walk of labels + value
+ }
+ sb.append("]}");
+ serializedRecords.add(sb.toString());
+}
+```
+
+`appendSampleJson`, `appendMeterEntityJson`, `appendBucketedValuesJson`,
+`appendLALOutputBuilderJson` (per known subclass), `appendISourceJson`
+(per scope) are all hand-written walks of the artifact's fields. Each
+artifact type has a known shape; we don't need to discover fields at
+runtime.
+
+For `LALOutputBuilder` subclasses (`LogBuilder`, `EnvoyAccessLogBuilder`,
+`DatabaseSlowStatementBuilder`, `SampledTraceBuilder`, custom subclasses
+contributed by `LALSourceTypeProvider`), the recorder dispatches via a
+small `Map<Class<? extends LALOutputBuilder>, BuilderSerializer>`
+populated at `dsl-debugging`'s startup. Custom builders contribute their
+own serializer through the `LALSourceTypeProvider` SPI alongside their
+existing `outputType()` contribution. If no serializer is registered the
+recorder falls back to a generic Java-bean walk that uses
+`PropertyDescriptor` reflection — only on this fallback path, not
+hot-path.
+
+##### Why no reflection on the hot path
+
+A `Jackson ObjectMapper.writeValueAsString(family)` would be 5-10× slower
+than the hand-written walk for `SampleFamily` because it would walk
+Sample's reflection metadata per call. With statement-level LAL active
+on a busy rule (8 records × 15 statements × 60s window), that adds up to
+visible CPU. Hand-written serializers keep us at ~500-1000 ns per
+capture, which is the active-path budget validated by the JMH active
+bench.
+
+##### Per-rule-instance `GateHolder` — codegen mechanics
+
+Every generated rule artifact gets one `GateHolder` field constructed
+in the class's constructor (or constructor-like initializer for
+dispatcher metric fields). The holder's `contentHash` is supplied as a
+constructor argument and stored in a `final` field, so it is fixed
+for that holder's entire lifetime:
+
+```java
+// MAL / LAL — instance field on the rule's singleton runtime class:
+public final GateHolder DEBUG = new GateHolder(/* contentHash = */ "7c3a91…");
+
+// OAL — per-metric instance field on the dispatcher singleton:
+public final GateHolder gate_endpoint_cpm = new GateHolder("8a21f0…");
+```
+
+The `contentHash` literal is computed at codegen time (SHA-256 over the
+content bytes the codegen consumed) and embedded directly in the
+generated bytecode's constant pool. There is no separate registry of
+holders, no `Map<RuleKey, GateHolder>` to dedup or replace, no
+`computeIfAbsent` semantics — each generated class instance simply
+constructs its own holder once.
+
+Session-install code finds the holder by asking each DSL's existing
+live-artifact registry (`MalExpression` / `LalExpression` /
+dispatcher singleton) for the rule and calling `.debugHolder(...)` —
+typed method, not reflection. Mutation goes through the holder's typed
+interface methods (`addRecorder`, `removeRecorder`); the gate
+transitions are encapsulated inside those methods, so callers can't
+forget to flip the gate. **No `Class.getField(...)`, no
+`setBoolean(null, ...)`, no reflection anywhere on the install /
+uninstall path.**
+
+Per-rule holder access on the hot path is the cheapest possible flag:
+a single `final` reference load + one volatile boolean read + branch.
+The `final` `DEBUG` reference is JIT-inlinable across the receiver's
+hot loop; the volatile read on `gate` is the actual gate check.
+
+##### Net summary
+
+- **Codegen emits**: probe call sites + `public final GateHolder DEBUG = new GateHolder("...sha...")` (or `gate_<metric> = new GateHolder(...)` per metric, for OAL dispatchers) + the `RuleKey ruleKey()` accessor. Hard-coded
+ bytecode literals; nothing computed at runtime.
+- **`{MAL,LAL,OAL}Debug` static methods**: hand-written probes. No
+ reflection.
+- **`DebugRecorderImpl` per-DSL appenders**: hand-written JSON
+ serializers per artifact type. No reflection on the hot path; only
+ fallback for unknown LAL output-builder subclasses.
+- **All allocation on the active path**: one `StringBuilder` per
+ capture (recycled? — no, fresh per capture; the per-capture lifetime
+ is short and ZGC/G1 handle it efficiently).
+
+#### 3.8 Interaction with rule changes — content-hash stamping
+
+Runtime-rule hot-update can replace a rule's class while a debug session
+is active. The captured `sourceLine` / `sourceText` describe the **rule
+content as of the class that produced them** — once the class swaps, a
+later record's source positions refer to a different content. The UI
+needs to know which content each record describes so it can render the
+right YAML alongside.
+
+The capture **carries the rule's content hash directly**, computed by
+the DSL engine at content-load time. The hash is **not persisted
+anywhere** — not in the runtime-rule DB row, not in any registry on
+disk. It lives only in memory: in the `GateHolder`, in the generated
+class's constant pool (as a String literal), and in the captured
+records that quote it. Storing it persistently would just denormalize
+content that's already in hand whenever it matters.
+
+Static rules loaded from disk at boot have no DB row at all — their
+content is read once, hashed in memory, and the hash flows into the
+generated class. Runtime-managed rules have a DB row carrying the rule
+content (status, content bytes, updateTime); the runtime-rule applier
+already loads that content into memory before compiling, so the hash
+falls out for free from the same in-memory bytes the codegen consumes.
+Both paths converge on the same SHA-256 from the same content with no
+extra DB column required.
+
+##### Extended `GateHolder`
+
+The `GateHolder` introduced in the
+[Performance plan](#36-performance-plan--idle-path-effectively-free-active-path-bounded)
+grows one field:
+
+The `GateHolder` carries `contentHash` as a `final` constructor argument
+(see [the canonical `GateHolder` definition](#2-capture-mechanism--per-rule-instance-gateholder-no-threadlocal-no-shared-registry)).
+
+Codegen emits the constructor call at the rule's instance-field
+initializer:
+
+```java
+// MAL / LAL — generated rule class:
+public final GateHolder DEBUG = new GateHolder(/* contentHash = */ "7c3a91…");
+
+// OAL — generated dispatcher:
+public final GateHolder gate_endpoint_cpm = new GateHolder("8a21f0…");
+```
+
+`contentHash` is computed once per class generation:
+
+| Origin | Where the hash is computed |
+|-----------------------------|---------------------------------------------------------------------------------------------------------|
+| Static-disk rules at boot | the analyzer reads the YAML from disk; codegen hashes those bytes locally before generating the class |
+| Runtime-rule managed rules | the runtime-rule applier loads content from the DB; codegen hashes those bytes locally — same code path |
+
+Both paths feed the codegen the same content bytes; codegen runs
+`SHA-256` over them once and embeds the hex digest as a String literal
+in the constant pool of the generated class. The DB row stores **only
+the content** (plus status + updateTime); the hash is recomputed
+whenever the content is loaded into memory, never read from a stored
+column.
+
+The runtime-rule plugin's `/runtime/rule/list` response includes a
+`contentHash` field for operator visibility — that response field is
+computed on demand from the in-memory content the plugin already has,
+not read from a persisted column. Same convention here: derive on
+load, don't store.
+
+##### What the recorder writes
+
+Every captured record reads `DEBUG.contentHash` at append time and stores
+it in the per-record JSON:
+
+```json
+{
+ "id": "r17",
+ "contentHash": "7c3a91…",
+ "stages": [
+ {"sourceLine": 7, "sourceText": "tagEqual('region', 'us-east-1')", "samples": [...]},
+ {"sourceLine": 7, "sourceText": "* 100", "samples": [...]}
+ ]
+}
+```
+
+A `volatile` field read is one cheap load; this is added to the active
+path's per-capture cost (already 500-2000 ns), not the idle path.
+
+##### What the session payload returns
+
+Top-level alongside `nodes[]`, the session response includes a
+`ruleSnapshots` map keyed by content hash:
+
+```json
+{
+ "sessionId": "d-abc",
+ "ruleSnapshots": {
+ "7c3a91…": {
+ "capturedFirstAt": 1745329200000,
+ "content": "filter: ...\nexpSuffix: ...\nmetricsRules:\n - ..."
+ },
+ "b1d402…": {
+ "capturedFirstAt": 1745329485000,
+ "content": "..."
+ }
+ },
+ "nodes": [...]
+}
+```
+
+Snapshots are populated by the `DebugSessionRegistry` the first time it
+sees a record carrying a hash it hasn't recorded yet. The content for a
+snapshot comes from the rule's source — runtime-rule's DB row for
+managed rules, the in-memory disk-read for static rules — neither of
+which is on the hot path.
+
+##### One hash, three places it surfaces — all the same value
+
+The same `contentHash` appears in three places, computed by the same
+SHA-256 over the same content bytes:
+
+| Place | Field | Purpose |
+|------------------------------------------------------------------------|------------------------------------------------|---------------------------------------------------------------|
+| Each captured record in a debug-session response | `record.contentHash` | Identify which rule version emitted this record |
+| `ruleSnapshots[hash]` in the same response | the map key + `.content` | Carry the rule text that matches `record.contentHash` |
+| Rule-content management APIs (existing `/runtime/rule/list`, new `/runtime/oal/files`, etc.) | `contentHash` field on each rule | Same identifier the UI uses to correlate captures with rules |
+
+The UI's invariant: `record.contentHash == ruleSnapshots[h].contentHash
+== /runtime/.../list.contentHash` for the same applied content. UI
+correlation is "look up by SHA prefix"; no parallel identifier scheme
+to map across.
+
+##### How hot-update interacts
+
+| Runtime-rule action | Holder state | Captured records |
+|------------------------------------|---------------------------------------------|----------------------------------------------------|
+| Filter-only update (same identity) | new `<clinit>` writes new `contentHash`; gate stays as-is | Pre-swap records keep their original hash; post-swap records carry the new hash |
+| Structural update | same — new hash on the holder | Records before vs. after the swap reference different snapshots; UI labels the boundary |
+| LAL block ↔ statement recompile | content unchanged — `contentHash` re-computes to the same SHA | All records carry one hash; `ruleSnapshots` has one entry |
+| Inactivate / delete | holder removed; sessions auto-terminate | Final session payload still has all snapshots already captured |
+| Static-only rule (no runtime-rule) | hash set once at boot; no further swaps | Single-hash session, single snapshot |
+
+##### Why content-hash, not a counter
+
+- **Static rules have no DB-side revision number.** Computing a SHA at
+ boot is the only honest identity the recorder can stamp.
+- **Idempotence.** A re-apply with byte-identical content produces the
+ same SHA → same `ruleSnapshots` entry → no spurious "rule changed"
+ boundary in the UI when nothing changed.
+- **Cluster agreement is guaranteed by the algorithm, not by
+ coordination.** Hashing rules:
+ - Algorithm: **SHA-256** over the rule's content bytes, hex-encoded
+ lowercase (64 chars).
+ - Byte source: the **exact bytes the DSL engine compiled** —
+ UTF-8-decoded YAML/`.oal` text from disk for static rules, the
+ UTF-8 bytes from the runtime-rule DB row for managed rules. **No
+ normalization** (no whitespace stripping, no line-ending rewrites,
+ no comment removal). Two nodes that compile the same content bytes
+ deterministically produce the same digest.
+ - Convergence path: runtime-rule's reconciler already converges
+ content cluster-wide within one tick (≤ 30 s). Within that
+ convergence window peers may briefly carry different content for
+ the same rule — and therefore different hashes. The session
+ payload's `nodes[]` array shows each peer's hash explicitly so the
+ UI sees the divergence honestly during the window; after
+ convergence all peers' captures stamp the same hash.
+ - Cross-version cluster: if OAP binaries differ across nodes, their
+ shipped static-rule bytes can differ, producing different hashes
+ for the "same" static rule. This is the same issue runtime-rule
+ documents as out of scope; deployment discipline is the fix.
+- **No counter to wrap, drift, or be surprising.** The hash IS the
+ identifier.
+
+#### 3.9 Permanent injection disable — boot-time bytecode opt-out
+
+This flag is **additive** to the existing runtime controls — not a
+replacement. Three orthogonal layers govern whether captures can run, in
+order of escalating strength:
+
+| Layer | Type | What it controls |
+|------------------------------------------------|-----------------------------------|-------------------------------------------------------------------------------------------|
+| `admin-server.enabled` | runtime (require restart to load) | Whether the admin HTTP server runs at all. If false, no REST endpoints exist. |
+| `dsl-debugging.enabled` | runtime | Whether the dsl-debugging module installs. If false, sessions can't be created. |
+| Per-rule `GateHolder.gate` | runtime | Whether captures fire on this rule for an active session. Flipped on session install/end. |
+| **`dsl-debugging.injectionEnabled` (NEW)** | **boot-time codegen** | **Whether probe call sites are emitted into generated bytecode at all.** |
+
+The first three already exist (or are introduced earlier in this SWIP)
+and remain valid — they handle the normal "operator pauses or disables
+the feature" cases without forcing a restart or a re-codegen. The fourth
+is what's new in this section: a stronger, more permanent guarantee for
+deployments that want **no probe call sites in the bytecode at all**.
+
+Why such a flag is useful even when the runtime flags exist:
+
+- High-assurance / regulated environments where any debug
+ instrumentation in production-loaded bytecode is a no-go regardless
+ of whether it would currently fire.
+- Defense in depth — even an admin who somehow flipped a `GateHolder`
+ cannot trigger captures because there are no call sites to fire.
+- Performance-paranoid sites that want absolute zero overhead past JIT
+ elimination (the `~1 ns volatile load` of the runtime-gate idle path
+ is not "absolute zero").
+
+A boot-time configuration flag governs this:
+
+```yaml
+dsl-debugging:
+ default:
+ # When false, the DSL codegens emit MAL/LAL/OAL classes WITHOUT any
+ # MALDebug / LALDebug / OALDebug call sites and WITHOUT the GateHolder static
+ # field. Generated bytecode is byte-identical to a build without
+ # SWIP-13. Default is true (capability preserved). Once set false at
+ # boot, debug captures cannot be enabled until restart with the flag
+ # back to true.
+ injectionEnabled: ${SW_DSL_DEBUGGING_INJECTION_ENABLED:true}
+```
+
+**What changes when `injectionEnabled: false`:**
+
+| Aspect | `injectionEnabled: true` (default) | `injectionEnabled: false` |
+|-------------------------------------------------------|-------------------------------------------------------------|--------------------------------------------------------------------|
+| Generated class — chain-stage call sites | `if (DEBUG.gate) MALDebug.captureStage(...)` | absent — line not emitted |
+| Generated class — `GateHolder DEBUG` static field | present, registered via `<clinit>` | absent |
+| live-artifact lookup (`liveHolderFor(key)`) | resolves to a real holder | unused — feature isn't installed |
+| `DebugSessionRegistry.install(...)` (mutates holder) | normal install path | `DebugSessionRegistry` is unbound; install rejected with 503 |
+| `POST /dsl-debugging/session` | accepts and starts a session | rejects with `503 injection_disabled` and a body explaining why |
+| Existing telemetry / data-path behaviour | identical | identical |
+
+**The flag is boot-time only, by design.** Flipping it at runtime would
+require regenerating every MAL/LAL/OAL class to either add or strip the
+probe call sites, and the runtime-rule reload infrastructure works on
+content changes, not on codegen-flag changes. Operators changing the
+flag must restart OAP for it to take effect. This is enforced by the
+provider: at startup the flag is read once into a `final boolean
+injectionEnabled` and threaded into every DSL generator's `GenCtx`. No
+hot-update path exists, no API mutates it.
+
+##### Status visibility — `GET /dsl-debugging/status`
+
+A new admin endpoint on `admin-server` exposes the current posture so
+operators / CI / compliance auditors can confirm a node's state without
+parsing YAML:
+
+| Method | Path | Effect |
+|--------|----------------------------|--------------------------------------------------------|
+| GET | `/dsl-debugging/status` | Return the current injection + session-runtime state |
+
+Response shape:
+
+```json
+{
+ "adminHostEnabled": true,
+ "dslDebuggingEnabled": true,
+ "injectionEnabled": false,
+ "injectionEnabledSource": "SW_DSL_DEBUGGING_INJECTION_ENABLED=false (boot)",
+ "sessionsAcceptingNewRequests": false,
+ "activeSessions": 0,
+ "maxActiveSessions": 200,
+ "ruleClassesWithProbes": 0,
+ "ruleClassesTotal": 87
+}
+```
+
+Field semantics:
+
+- `injectionEnabled` — the boot-resolved value of the flag. Always
+ reflects what the codegen actually did.
+- `injectionEnabledSource` — human-readable provenance: which env var or
+ YAML key set the value, and that it was resolved at boot.
+- `sessionsAcceptingNewRequests` — `false` whenever **any** of the four
+ layers above (`admin-server.enabled`, `dsl-debugging.enabled`, all rule
+ `GateHolder` references, or `injectionEnabled`) puts the system into
+ a state that can't actually capture. The flag composes the runtime
+ posture into a single yes/no for clients.
+- `ruleClassesWithProbes` / `ruleClassesTotal` — quick visual confirmation
+ that the codegen actually applied the flag. With
+ `injectionEnabled: false` the first count is 0; with `true` it equals
+ the total. Disagreement signals a bug or partial reload.
+
+`GET /dsl-debugging/status` is read-only, returns immediately, has no
+session-cap interaction, and is safe to scrape at high frequency from
+monitoring tools.
+
+##### Cluster behaviour
+
+`injectionEnabled` is per-node — it's a boot-time codegen choice, not a
+clustered configuration. A cluster running with mixed
+`injectionEnabled` values has some nodes that can capture and others
+that cannot. The `nodes[]` array in a session response surfaces this
+honestly: nodes with `injectionEnabled: false` contribute
+`{status: "injection_disabled"}` slices instead of records. Operators
+who need uniform behaviour set the flag uniformly across the cluster
+(deployment discipline; the SWIP doesn't synchronise it).
+
+#### 3.10 Security — out of scope; delegated to the deployment
+
+The capture API is read-side and can expose sensitive payloads (raw
+log bodies, parsed maps, extracted fields) when the operator debugs
+the LAL surface. **Securing access is out of scope for this SWIP.**
+`admin-server` ships unauthenticated this iteration; the deployment is
+responsible for putting it behind a sidecar / gateway / IP allow-list.
+SWIP-13 does not introduce auth, redaction, or masking primitives —
+those concerns live in the dedicated *Admin API* documentation
+(see [`docs/en/setup/backend/admin-api/`](../setup/backend/admin-api/readme.md)),
+which carries the security notice for `admin-server`, the runtime-rule
+update API, and the DSL debug API together. The notice is the single
+canonical place operators should read before opening the admin port to
+their cluster.
+
+### 4. REST surfaces on the shared `admin-server`
+
+`admin-server` is the runtime / on-demand admin surface. It hosts exactly
+two concerns in this SWIP — runtime-rule (existing, migrated onto the
+shared server) and DSL debugging (new). They share the same Armeria
+server, the same admin port (default 17128), and the same security
+posture (no auth in this release, gateway-protect, ships disabled by
+default):
+
+| Subsection | Purpose | Origin |
+|----------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|-------------------------------------------------------------------------|
+| [4.1 OAL read-only management API](#41-oal-read-only-management-api--the-catalog-the-debugger-picks-from) | List loaded `.oal` files / rules so the OAL debugger has a picker | NEW — this SWIP |
+| [4.2 DSL debug session API](#42-dsl-debug-session-api) | Start / poll / stop debug sessions across MAL / LAL / OAL | NEW — this SWIP |
+| existing `/runtime/rule/*` (migrated) | Runtime-rule hot-update endpoints, hosted on admin-server post-migration | MIGRATED — runtime-rule plugin's own HTTP server is removed |
+
+**Not on `admin-server`** (intentionally): the `status-query-plugin`
+endpoints `GET /status/config/ttl` and `GET /debugging/config/dump`,
+plus all `/debugging/query/*` step-through helpers, stay where they are
+on the query port (default 12800). They are released APIs serving a
+read-only telemetry-inspection audience and we do not move them. The
+`admin-server` URI namespace is reserved for write/admin endpoints
+(runtime-rule writes, dsl-debugging session control). Operators who
+want config inspection in addition to admin operations open both ports
+on their gateway, exactly as today.
+
+#### 4.1 OAL read-only management API — the catalog the debugger picks from
+
+Today MAL and LAL rules are runtime-managed (the runtime-rule plugin
+exposes `/runtime/rule/list`), but **OAL rules are not** — they are loaded
+from `oap-server/server-starter/src/main/resources/oal/*.oal` at boot and
+have no runtime listing API. That gap blocks the OAL debugger: the
+operator has to pick "one OAL rule line" to sample, and there is currently
+no way for the UI or `swctl` to discover what rules exist without reading
+the source `.oal` files off disk.
+
+This SWIP closes that gap with a **read-only** OAL management API.
+Read-only because OAL hot-update (add/update/inactivate/delete at runtime)
+is a much larger feature with its own classloader-isolation, alarm-reset,
+and cluster-coordination concerns — that is explicitly **out of scope**
+here, scoped for a future SWIP. What lands now is just the listing,
+sufficient to drive the OAL debugger's rule picker.
+
+The API lives in a new small module `oal-management` (or as a
+sub-package inside `oal-rt` — the implementer picks based on dependency
+hygiene), registers its handler on `admin-server`, and is structured so that
+adding write endpoints later is purely additive.
+
+| Method | Path | Effect |
+|--------|---------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------|
+| GET | `/runtime/oal/files` | List loaded `.oal` files. One row per file: `{ name, path, ruleCount, status, contentHash }` |
+| GET | `/runtime/oal/files/{name}` | Single file detail: `{ name, path, content, rules: [...] }` where `content` is the raw `.oal` text and `rules` are the parsed rule definitions |
+| GET | `/runtime/oal/rules` | Flat list of every rule across every file: `[{ file, ruleName, line, sourceScope, expression, function, filters }, ...]` |
+| GET | `/runtime/oal/rules/{ruleName}` | Single rule detail: `{ file, ruleName, line, sourceScope, expression, function, filters, persistedMetricName }` |
+
+Response shape per rule:
+
+```json
+{
+ "file": "core",
+ "ruleName": "endpoint_cpm",
+ "line": 14,
+ "sourceScope": "Endpoint",
+ "expression": "endpoint_cpm = from(Endpoint.*).cpm();",
+ "function": "cpm",
+ "filters": [],
+ "persistedMetricName": "endpoint_cpm",
+ "contentHash": "8a21f0…"
+}
+```
+
+`contentHash` is the same SHA-256 the debugger stamps on captured records
+([content-hash stamping](#38-interaction-with-rule-changes--content-hash-stamping)) — UI
+matches `record.contentHash` against the `contentHash` returned here to
+fetch the right content for rendering. The hash is computed on demand
+from the in-memory file content the OAL engine already holds; not stored
+in the DB.
+
+The fields come from the existing `oal-rt` parsing pipeline (parser →
+`MetricDefinition` → `MetricDefinitionEnricher` → `CodeGenModel`),
+extended with `contentHash` carried alongside on the
+`OALRuleRegistry.OalFileSnapshot` record (defined below).
+
+###### `OALRuleRegistry` — explicit retention contract
+
+**Today's loader does not retain the parsed structures**:
+[`OALEngineV2.start`](../../../oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/OALEngineV2.java)
+parses, enriches, generates classes, and lets the local `OALScriptParserV2`
+and `List<CodeGenModel>` go out of scope.
+[`OALEngineLoaderService`](../../../oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/oal/rt/OALEngineLoaderService.java)
+only stores the `OALDefine` set; the parsed AST is discarded after class
+generation. Naively re-parsing on each REST call would work but is fragile
+(two engines see different shapes if one diverges) and slow.
+
+This SWIP introduces a small new service in `server-core` that the
+loader populates at boot:
+
+```java
+package org.apache.skywalking.oap.server.core.oal.rt;
+
+public interface OALRuleRegistry extends Service {
+ /** Snapshot loaded at boot; updated only by future OAL hot-update. */
+ List<OalFileSnapshot> files();
+ Optional<OalFileSnapshot> file(String fileName);
+ Optional<OalRuleSnapshot> rule(String fileName, String ruleName);
+
+ record OalFileSnapshot(
+ String name, // "core"
+ String path, // "core.oal"
+ String content, // raw .oal text
+ String contentHash, // SHA-256 of `content` bytes — matches captures (see content-hash stamping section)
+ List<OalRuleSnapshot> rules,
+ Status status // LOADED | DISABLED | COMPILE_FAILED
+ ) {}
+
+ record OalRuleSnapshot(
+ String file, String ruleName, int line,
+ String sourceScope, // e.g. "Endpoint"
+ String expression, // verbatim line(s)
+ String function, // "cpm" | "longAvg" | "apdex" | …
+ List<FilterSnapshot> filters,
+ String persistedMetricName,
+ String contentHash // copy of file's contentHash for convenience
+ ) {}
+}
+```
+
+`OALEngineV2.start()` calls `registry.put(file, snapshot)` after class
+generation succeeds. Memory cost is bounded by the OAL rule set (~200
+rules across ~12 files = a few hundred KiB of String + small objects);
+retained for the process lifetime.
+
+When OAL hot-update lands in a future SWIP, the same registry receives
+updates through the same `put()` path — the read API doesn't change.
+The DSLDebuggingProvider's REST handler reads from this registry; no
+re-parsing on the request path.
+
+##### How the debugger picks a rule
+
+OAL debugging requires the operator to pick `(file, ruleName)` because OAL
+captures sample source rows for **one** rule line, not the whole file. The
+flow is:
+
+1. UI calls `GET /runtime/oal/rules` to populate the picker.
+2. Operator picks a rule (e.g. `endpoint_cpm` from `core.oal`).
+3. UI calls `POST /dsl-debugging/session` with
+ `catalog: "oal"`, `name: "core"` (the file), `ruleName: "endpoint_cpm"`.
+4. Backend appends the recorder to that rule's `GateHolder.recorders`
+ array and flips its `gate` if it was the first session — only that
+ rule's `do<Metric>(source)` probes capture.
+
+For MAL and LAL the picker is fed by the existing
+`/runtime/rule/list` (since those catalogs are already runtime-managed).
+The OAL picker is the only one that needs this new API.
+
+##### Phase-0 scope reminder
+
+In this SWIP we ship **only the GET endpoints listed above**. POSTs that
+would mutate OAL state are deferred to a follow-up SWIP. The path prefix
+`/runtime/oal/*` is reserved so that future write endpoints (hot-update,
+inactivate, delete) can land without a URL break.
+
+#### 4.2 DSL debug session API
+
+REST routes live in the new `dsl-debugging` module and register on
+`admin-server`'s Armeria server. URI prefix is `/dsl-debugging/*` — clean
+and self-describing, no overload of the query port's existing
+`/debugging/*` prefix.
+
+| Method | Path | Body (optional JSON) | Effect |
+|--------|-----------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------|
+| POST | `/dsl-debugging/session?catalog=&name=&ruleName=&clientId=&granularity=` | `{ recordCap?, retentionMillis?, granularity?: "block" \| "statement" }` | Start session; **also stops any prior session bound to `clientId`**; broadcast install to peers; return `{ sessionId, createdAt, retentionDeadline, installed: {created, total}, peers[], priorCleanup }` |
+| GET | `/dsl-debugging/session/{id}` | — | Return aggregated payload (see [data contract](#1-the-data-contract-returned-to-the-ui)). 404 `session_not_found` if no node knows the id. |
+| POST | `/dsl-debugging/session/{id}/stop` | — | Force close; idempotent. |
+| GET | `/dsl-debugging/sessions` | — | List active sessions on this node (JSON). |
+| GET | `/dsl-debugging/status` | — | Module posture: `{module, phase, nodeId, injectionEnabled, activeSessions}`. |
+
+`clientId` is a stable identifier the client mints **once per debug
+context**. The UI decides what counts as a debug context — typically
+one per browser tab when the page hosts a single debug widget, or one
+per widget when the UI embeds several debug views on the same page.
+The CLI mints one per invocation. The server doesn't model the tab /
+page / widget distinction; it only enforces "one `clientId` →
+at most one active session, and a new POST replaces the prior". See
+[Session lifecycle, storage, and memory control](#5-session-lifecycle-storage-and-memory-control)
+for how it interacts with the lifecycle.
+
+For OAL sessions, `name` is the `.oal` file (without extension) and
+`ruleName` is the rule name picked from `/runtime/oal/rules`. For MAL and
+LAL sessions, `ruleName` is omitted and `(catalog, name)` matches a row in
+`/runtime/rule/list`.
+
+Response payload is shaped per-DSL exactly as the
+[data contract](#1-the-data-contract-returned-to-the-ui) describes; the
+receiving node merges its own slice with peer slices into one `nodes[]`
+array under the top-level shape so the UI can show "captured 4 of 6 nodes
+responded · oap-03 timed out".
+
+Security: `admin-server` inherits and centralises the security caveats already
+applied to the runtime-rule REST endpoints — no auth in this release, the
+host module ships **disabled by default**, must be gateway-protected, must
+not be exposed to the public internet. Runtime-rule, OAL management, and
+DSL-debugging routes are all gated by the same enable flag on `admin-server`.
+No new auth mechanism is introduced.
+
+#### `admin-server` — what the shared module provides
+
+```
+oap-server/server-admin/admin-server/ # NEW (under new server-admin/ parent)
+ src/main/java/.../admin/
+ AdminServerModule.java # ModuleDefine
+ AdminServerProvider.java # ModuleProvider — owns Armeria server, port, host, lifecycle
+ AdminRouteRegistry.java # Service — sub-modules register handler classes
+ AdminAuthDocumentation.java # one place to document the no-auth caveat
+```
+
+Sub-modules call `registry.register(handlerInstance)` on `prepare()`;
+`admin-server` aggregates them into one Armeria server bound at boot. No
+sub-module owns its own HTTP server.
+
+#### `dsl-debugging` — what the new feature module provides
+
+```
+oap-server/server-admin/dsl-debugging/ # NEW (sibling of admin-server under server-admin/)
+ src/main/java/.../dsldebugging/
+ DSLDebuggingModule.java # ModuleDefine
+ DSLDebuggingProvider.java # ModuleProvider — registers REST handler on admin-server
+ rest/DSLDebuggingHandler.java # Armeria @Get/@Post routes for /dsl-debugging/*
+ session/DebugSessionRegistry.java # Map<SessionId, DebugSession> + Map<ClientId, SessionId>
+ session/DebugSession.java # frozen List<String> + lifecycle states
+ session/DebugRecorderImpl.java # implements DebugRecorder, JSON-serialize-on-capture
+ bus/InstallDebugSessionRpc.java # cluster fan-out RPCs
+ bus/CollectDebugSamplesRpc.java
+ bus/StopDebugSessionRpc.java
+ bus/StopByClientIdRpc.java # cluster-scope clientId cleanup (load-balancer safe)
+```
+
+The marker `DebugRecorder` interface, `GateHolder`, `RuleKey`, and the
+OAL probe class `OALDebug` live in
+`server-core` (so the MAL/LAL/OAL generators can reference them without
+pulling the whole feature module — keeps the dependency graph clean).
+`dsl-debugging` provides the recorder implementation, registry, and REST
+host registration.
+
+#### `runtime-rule` migration — module relocation + HTTP server merge
+
+Two moves in one release:
+
+1. **Module relocation.** The existing runtime-rule plugin lives under
+ `oap-server/server-receiver-plugin/skywalking-runtime-rule-receiver-plugin/`
+ today, but it doesn't receive telemetry data — it serves an admin
+ surface. Under this SWIP the module relocates to
+ `oap-server/server-admin/runtime-rule/`, becoming a sibling of
+ `admin-server` and `dsl-debugging` under the new `server-admin/`
+ parent:
+
+ ```
+ oap-server/server-admin/ # NEW Maven sub-module group
+ admin-server/ # shared Armeria HTTP server
+ runtime-rule/ # MOVED from server-receiver-plugin/
+ dsl-debugging/ # NEW DSL debugging feature
+ ```
+
+ The selector key in `application.yml` stays as `receiver-runtime-rule:`
+ for backward compatibility (operators upgrading don't have to rename
+ the YAML block); the Java package stays as
+ `org.apache.skywalking.oap.server.receiver.runtimerule.*` — only the
+ Maven module path changes. A future cleanup can rename both, but
+ that's not in scope here.
+
+2. **HTTP server merge.** The runtime-rule plugin's own embedded
+ Armeria server is removed; its REST handlers register on
+ `admin-server`'s shared server. URL paths stay the same:
+
+ ```
+ /runtime/rule/addOrUpdate
+ /runtime/rule/fix
+ /runtime/rule/inactivate
+ /runtime/rule/delete
+ /runtime/rule/list
+ /runtime/rule/dump
+ ```
+
+ Existing CLI scripts (`runtime-rule.sh`) and operator muscle memory
+ keep working — same routes, same response codes, same body semantics,
+ just hosted on the shared HTTP server.
+
+Both moves happen in Phase 0 (see [Phased delivery](#10-phased-delivery)) so
+`server-admin/` lands as a coherent group rather than dribbling in
+piece by piece.
+
+### 5. Session lifecycle, storage, and memory control
+
+Captured data is **per-session, immutable once captured, and never overwritten**.
+Operator stability comes first: once a sample lands in the session, repeated polls
+return the same bytes until the session is explicitly stopped or hits its timeout.
+There is no ring buffer, no drop-oldest — when the cap is reached, the session
+simply stops accepting new samples.
+
+#### Lifecycle states
+
+```
+ [absent] ──POST /dsl-debugging/session──► CAPTURING ──cap-or-window──► CAPTURED
+ │ │ │
+ │ └──manual /stop───────────┤
+ │ │
+ └──── timeout ────────────► EXPIRED ──► [absent]
+ ▲
+ CAPTURED + ttl
+ ── new POST with same clientId ⇒ prior session forced to EXPIRED ──
+```
+
+- **CAPTURING** — session is installed; capture hooks append samples. Polling is
+ allowed and returns the partial payload accumulated so far (`status:
+ "capturing"`).
+- **CAPTURED** — capture is complete (record cap hit, window elapsed, or operator
+ called `/stop` mid-capture). The payload is now frozen. Polling returns the
+ same bytes on every call (`status: "captured"`).
+- **EXPIRED** — the session's timeout fired. Stored payload is dropped from
+ memory; further polls return 404. Operator must start a fresh session.
+
+Three events terminate a session and free its memory:
+
+1. **Manual trigger** — `POST /dsl-debugging/session/{id}/stop` (mid-capture or
+ post-capture). The session moves directly to EXPIRED on the next sweep —
+ manual stop is interpreted as "I have what I need, throw it away".
+2. **Implicit cleanup on a new sampling request — cluster-scope.**
+ When a `POST /dsl-debugging/session` arrives carrying a `clientId`,
+ the receiving node first **broadcasts a `StopByClientId(clientId)`
+ RPC to every peer** (including itself) and waits for ack within the
+ per-peer fanout deadline. Each node looks up `clientId` in its own
+ local `Map<ClientId, SessionId>`; if a prior session is bound, the
+ peer terminates it locally (decrements the rule's recorder
+ ref-count, drops its payload). Once the broadcast settles (or the
+ per-peer deadline expires), the receiving node allocates the new
+ session and runs the normal `InstallDebugSession` fan-out.
+
+ The cluster broadcast is **load-balancer safe**: because every node
+ receives `StopByClientId(clientId)`, it doesn't matter which node
+ the load balancer routes the next POST to — the previous session
+ bound to that `clientId` (wherever it lives) gets terminated before
+ the new one is allocated. Without this, a load balancer that bounced
+ between nodes would leave the prior session alive on its original
+ node until retention timeout, and the active-session-cap math would
+ silently double-count.
+
+ Implementation note: the broadcast is **best-effort** with a short
+ deadline (default 2 s, same as runtime-rule's `Suspend` fanout). A
+ peer that doesn't ack within the deadline is logged in the
+ response's `peers[]` array. **No durable management-storage marker
+ is introduced** — the prior session on the missed peer self-cleans
+ up via the same retention timeout that handles forgotten sessions
+ in general (default 10 min after CAPTURED). The trade is honest:
+ under partition the active-session count on the missed peer is
+ slightly inflated (by the prior session) until retention elapses,
+ but the peer's own `maxActiveSessions: 200` cap protects it from
+ accumulating unbounded stragglers.
+3. **Timeout** — see the cap table below.
+
+A session that has finished capturing (`CAPTURED`) is **not** auto-discarded just
+because the capture window ended. It lingers until either the operator stops it
+or the timeout fires, so the UI can poll without racing against a fast
+disappearance.
+
+#### Caps
+
+| Cap | Default | Mechanism |
+|--------------------------------------|--------------------------------------|------------------------------------------------------------------------|
+| **Max active sessions per node** | **200** | hard ceiling; `POST /dsl-debugging/session` returns 429 too_many_sessions when full |
+| **Records per session** | **default 1000, hard cap 10000** | recorder stops appending once the count is hit and moves the session to CAPTURED. Out-of-range request returns 400 invalid_limits. |
+| Capture window (retention) | default 5 min, hard cap 1 hour | per-session retention timeout; sweeper drops the payload at window end |
+| Capture-call cost when idle | one volatile-load + branch (gate) | JIT eliminates the call site when gate is false |
+
+The capture surface is bounded by **two** dimensions only: how many
+sessions can be active concurrently, and how many records each session
+can retain. There is intentionally **no per-session byte cap and no
+per-record byte budget** — operators sizing for memory look at
+`maxActiveSessions × recordCap` and the per-DSL average payload size.
+JSON serialization for the wire happens at probe time so polls return
+the same bytes every time and the retention sweeper can free them on a
+fixed schedule, but the byte total is reported (not enforced) so
+operators can verify their own budget assumptions against real
+captures.
+
+#### Why 200 active sessions is the right ceiling
+
+200 sessions is a defensible upper bound for an OAP node. With the
+hard `recordCap = 10000` and a typical per-record JSON payload of
+~10 KiB (MAL is the largest of the three), the worst-case footprint
+across all active sessions on one node is roughly:
+
+```
+worst-case heap ≈ MAX_ACTIVE_SESSIONS × MAX_RECORD_CAP × per-record-bytes
+ ≈ 200 × 10 000 × 10 KiB ≈ 20 GiB (theoretical)
+```
+
+That's the worst-case product; realistic usage is one to two orders of
+magnitude below because most captures stop on retention (5 min default)
+or operator stop long before hitting `recordCap`, and most records are
+several KiB rather than the worst-case 10. The 200 default leaves
+headroom for many concurrent operators (one debug context = one
+sessionId; the UI maintains a single session per debug widget and
+reuses it for polls) without letting a runaway script exhaust the
+heap. Operators tuning for tighter heap budgets stop sessions sooner
+or lower per-session `recordCap` in the request body — the per-node
+ceiling itself is fixed by `MAX_ACTIVE_SESSIONS`.
+
+The session-id is the unit of accounting; the **clientId** is the unit of
+deduplication. The UI is expected to:
+
+- Mint a stable `clientId` per debug context — typically one per browser
+ tab when the page hosts a single debug widget; one per widget when
+ the page embeds multiple. UUID stored in `sessionStorage` (per-tab) or
+ in a per-widget keyed map on the page.
+- Send `clientId` on every `POST /dsl-debugging/session`. The server cleans up
+ the prior session for that `clientId` before allocating a new one, so a
+ forgetful operator who clicks Start sampling repeatedly does not exhaust the
+ active-session ceiling — they hold one slot at a time.
+- Poll the returned `sessionId` until the user navigates away or hits **Stop**.
+- Send `POST /dsl-debugging/session/{id}/stop` on tab close (best-effort
+ `beforeunload`).
+
+A future CLI follows the same pattern: `start` allocates one session under a
+stable client ID, `tail` reuses it, `stop` releases it. Stale sessions that
+the client forgot to stop are reaped by the retention timeout, and any new
+`start` from the same client implicitly cleans up the prior one regardless.
+
+#### Configuration block — `application.yml`
+
+Two top-level blocks: `admin-server` (the shared HTTP server) and
+`dsl-debugging` (the feature). `admin-server` is also where the existing
+runtime-rule plugin reads its host/port now — the HTTP-server keys move
+out of `receiver-runtime-rule:` and into `admin-server:`. The runtime-rule
+plugin keeps only its non-HTTP knobs (reconciler interval, self-heal
+threshold).
+
+The structure is **one shared block + two feature blocks on top of it**:
+
+```
+admin-server: ← shared HTTP server (host, port, security posture)
+ ↑
+ ├─ receiver-runtime-rule: ← feature 1 (DSL hot-update). Routes register on admin-server.
+ └─ dsl-debugging: ← feature 2 (DSL debug sessions). Routes register on admin-server.
+```
+
+Either feature requires `admin-server.selector=default`. **All three
+selectors default to empty (disabled).** Operators explicitly opt in
+to whichever surface they want; out-of-the-box OAP starts with no
+admin port, no debug feature, no runtime-rule.
+
+The "fail fast if a feature is enabled without admin-server" guarantee
+uses the **OAP module system's existing `requiredModules()`
+mechanism** — no new SWIP machinery needed:
+
+```java
+// in DSLDebuggingProvider:
+@Override public String[] requiredModules() {
+ return new String[] { CoreModule.NAME, AdminServerModule.NAME };
+}
+
+// in RuntimeRuleProvider (after Phase 0 migration):
+@Override public String[] requiredModules() {
+ return new String[] { CoreModule.NAME, AdminServerModule.NAME, ... };
+}
+```
+
+`ModuleManager.bootstrap()` walks every loaded provider, resolves
+`requiredModules()` against the provider set, and fails the OAP boot
+with a `ModuleNotFoundException` if any required module is absent. So:
+
+| Operator config | Boot result |
+|------------------------------------------------------------------------|------------------------------------------------------------------------------|
+| All three selectors empty (default) | OAP starts; no admin port; no debug; no runtime-rule. |
+| `admin-server=default`, `dsl-debugging=` empty | OAP starts; admin-server runs (with no consumers); idle. |
+| `admin-server=default`, `dsl-debugging=default` | OAP starts; both bind; routes register on admin-server. |
+| `admin-server=` empty, `dsl-debugging=default` | **Boot fails fast** with `ModuleNotFoundException: admin-server` — operator sees the error and adds `SW_ADMIN_SERVER=default`. |
+| `admin-server=` empty, `receiver-runtime-rule=default` | Same — boot fails fast with the same exception path. |
+
+This matches every other receiver / admin module in OAP that depends
+on shared infrastructure (every receiver depends on `CoreModule` the
+same way; every storage-using module depends on `StorageModule`, etc.).
+SWIP-13 is reusing the established mechanism, not introducing a new one.
+
+```yaml
+# Shared HTTP server for admin / write APIs (runtime-rule, dsl-debugging).
+# DISABLED BY DEFAULT. Enable when any consumer is needed.
+# SECURITY NOTICE: no authentication this iteration — gateway-protect
+# with IP allow-lists and never expose to the public internet.
+admin-server:
+ selector: ${SW_ADMIN_SERVER:-}
+ default:
+ host: ${SW_ADMIN_SERVER_HOST:0.0.0.0}
+ port: ${SW_ADMIN_SERVER_PORT:17128}
+ contextPath: ${SW_ADMIN_SERVER_CONTEXT_PATH:/}
+ idleTimeOut: ${SW_ADMIN_SERVER_IDLE_TIMEOUT:30000}
+ acceptQueueSize: ${SW_ADMIN_SERVER_QUEUE_SIZE:0}
+ httpMaxRequestHeaderSize: ${SW_ADMIN_SERVER_HTTP_MAX_REQUEST_HEADER_SIZE:8192}
+
+# Runtime-rule plugin — non-HTTP knobs only after SWIP-13.
+# HTTP server config moves to `admin-server:` above. Enabling
+# receiver-runtime-rule requires admin-server to also be enabled
+# (selector=default); startup fails with a clear error otherwise.
+receiver-runtime-rule:
+ selector: ${SW_RECEIVER_RUNTIME_RULE:-}
+ default:
+ refreshRulesPeriod: ${SW_RECEIVER_RUNTIME_RULE_REFRESH_RULES_PERIOD:30}
+ selfHealThresholdSeconds: ${SW_RECEIVER_RUNTIME_RULE_SELF_HEAL_THRESHOLD_SECONDS:60}
+
+dsl-debugging:
+ selector: ${SW_DSL_DEBUGGING:-}
+ default:
+ # Boot-time codegen switch — enable / disable per-rule probe call sites
+ # in the generated MAL / LAL / OAL bytecode. Default true once the
+ # module is enabled; set false to keep the REST surface but emit
+ # bytecode byte-identical to a build without SWIP-13.
+ injectionEnabled: ${SW_DSL_DEBUGGING_INJECTION_ENABLED:true}
+```
+
+The active-session ceiling and per-session record cap are NOT operator
+knobs — they are hard-coded SWIP contract values:
+
+- `MAX_ACTIVE_SESSIONS = 200` per node — `POST /dsl-debugging/session`
+ returns `429 too_many_sessions` once the count is reached. The
+ receiving node also runs the prior-session cleanup pass for the
+ supplied `clientId` BEFORE counting toward this ceiling, so a single
+ client repeatedly clicking Start sampling cannot itself trigger 429.
+- `MAX_RECORD_CAP = 10000` per session — request bodies asking for more
+ return `400 invalid_limits`. Session retention is similarly capped
+ at 1 hour (`MAX_RETENTION_MILLIS`).
+
+There is intentionally **no per-session byte cap and no structural
+char / sample / label sub-caps**. Operators sizing for memory pressure
+look at `MAX_ACTIVE_SESSIONS × recordCap × per-record-payload-size`.
+Per-DSL average payload size is reported as `totalBytes` on every GET
+response so operators can verify their budget against real captures.
+
+#### Storage location
+
+Per-session payloads live in a process-local `Map<SessionId, DebugSession>`
+inside the `dsl-debugging` module, alongside a sibling
+`Map<ClientId, SessionId>` index used by the implicit-cleanup path. **No
+persistence layer.** Sessions are lost on OAP restart by design — they are
+debugging snapshots, not durable artefacts. Operators who need to keep a
+result snapshot the response body client-side (the API already returns the
+full payload; the CLI's `tail` command saves it to a file).
+
+### 6. Cluster behaviour — best-effort fan-out (no cross-node merge)
+
+OTLP / log / native-trace data lands on whichever OAP receives the push, so a
+single-node debug session would only catch the slice of traffic that node owns.
+The debugger therefore broadcasts the session install to every peer **so each peer
+captures its own slice of L1 parsing**.
+
+Important: the fan-out is **not** an attempt to follow data across nodes. There is no
+L2 join in scope — every peer's slice is a self-contained "what arrived here, what
+the DSL did with it, what L1 the rule emitted on this node". The aggregator on the
+receiving node concatenates per-node slices into a `nodes[]` array; it does **not**
+merge or reconcile samples between nodes.
+
+The fan-out rides the **admin-internal gRPC bus** owned by
+`admin-server` (default port `17129`); `dsl-debugging` ships
+`DSLDebuggingClusterServiceImpl` and registers it alongside
+runtime-rule's `RuntimeRuleClusterServiceImpl` on the same admin-
+internal port. Four RPC types: `InstallDebugSession` /
+`StopDebugSession` / `CollectDebugSamples` / `StopByClientId`. Peer
+discovery is handled by `AdminClusterChannelManager`, which dials each
+peer's admin-internal port via the cluster module's existing peer
+registry. The default per-call timeout is configurable via
+`admin-server.internalCommunicationTimeout` (default `5000` ms).
+
+#### Install (POST `/dsl-debugging/session`)
+
+1. **Cluster-scope clientId cleanup first.** Receiving node sweeps any
+ locally-bound prior session for the same `clientId` and broadcasts
+ `StopByClientId(clientId)` to every peer with the configured
+ admin-internal-RPC deadline. Each peer looks up `clientId` in its
+ own session map; if found, terminates the bound session locally
+ (calls `session.boundHolder.removeRecorder`, drops payload, frees
+ active-session slot). The fan-out is best-effort — a missed peer
+ is logged in `peers[]`; its stale prior session for that
+ `clientId` self-cleans-up via retention timeout. The install does
+ not block on slow peers.
+2. Receiving node generates the new `sessionId` and computes
+ `expiresAt`, binds `clientId → sessionId` in its own local index.
+3. It iterates the sorted peer list (same view
+ `AdminClusterChannelManager` exposes to the runtime-rule
+ main-selection logic) and sends
+ `InstallDebugSession(sessionId, spec, expiresAt)` to each peer with
+ the same per-peer deadline.
+4. A peer that does not ack within the deadline is recorded as
+ `install_failed` for that peer and the fanout continues — no peer
+ can stall the operator.
+5. On the receiving node and on every peer that acked, the session is
+ installed by mutating the rule's `GateHolder` (`addRecorder(...)`);
+ receiver threads on that node read the holder directly.
+6. The receiving node returns `200 { sessionId, expiresAt,
+ replacedPriorId?: priorSessionId, peers: [{nodeId, ack:
+ ok|install_failed|timeout}] }`. The optional `replacedPriorId` makes
+ the cleanup observable to the client. If multiple nodes had a prior
+ session bound to the same `clientId` (impossible under correct
+ routing, but defensible if a load-balancer split caused one), every
+ stopped session is reported in the response so the operator can see
+ the clean-up footprint.
+
+If **no peer** acks (full partition), the session still runs on the receiving
+node; the response simply reports zero healthy peers. This is the documented
+"best effort" contract.
+
+#### Collect (GET `/dsl-debugging/session/{id}`)
+
+1. Receiving node sends `CollectDebugSamples(sessionId)` to every peer with a
+ short deadline (default 3 s).
+2. Each peer returns its local payload slice (or an empty slice + reason if
+ the session never installed there).
+3. The receiving node merges slices into the `nodes[]` array under the shape
+ from the [data contract](#1-the-data-contract-returned-to-the-ui) and
+ returns `200`. Peers that timed out are recorded as
+ `nodes[].status = "timeout"` rather than being omitted, so the UI shows
+ partial coverage honestly.
+
+#### Stop (POST `/dsl-debugging/session/{id}/stop`)
+
+Best-effort broadcast of `StopDebugSession(sessionId)`; missing acks are logged but
+do not block the response. The TTL sweeper is the backstop for a peer that missed
+the stop.
+
+#### Why fan-out and not a single main
+
+Unlike runtime-rule, debug sessions are not destructive — duplicate installs are
+idempotent (same `sessionId` ⇒ no-op), there is no DDL, and a peer that never gets
+the install simply contributes nothing to the result. There is no correctness reason
+to serialize through a single main, and the fan-out gives the operator visibility
+into traffic on every node.
+
+#### Consistency bounds
+
+| Event | Bound |
+|----------------------------------------------------|--------------------------------------------------------------------------------------------------------|
+| Healthy install | every reachable peer receives within fanout deadline (default 2 s) |
+| Peer partitioned during install | session simply does not capture there; reported in `peers[]` |
+| Peer crashes mid-session | session expires via retention timeout on remaining nodes; collect reports `node_unreachable` |
+| Operator polls collect during fanout | aggregator returns partial result; UI shows progress |
+| Two operators install concurrently (different `clientId`s) | independent `sessionId`s, no shared state, independent caps |
+| Same client's POSTs split across nodes by load balancer | new POST broadcasts `StopByClientId(clientId)` to every peer before allocating; reachable peers terminate the prior session immediately; a missed peer's stale session falls out via retention timeout (~10 min); no durable storage marker required |
+
+### 7. UI surface (recap)
+
+The Claude Design handoff already locked the three views:
+
+- **MAL** Waterfall · entity-scope selector · per-stage SampleFamily tables
+- **LAL** records-as-columns × blocks-as-rows grid · split `output·log` /
+ `output·metric` rows · per-block sampling toggle
+- **OAL** rule-name picker fed by `/runtime/oal/rules` · `(maxRows, windowSec)`
+ ▶ Start sampling button · 5-stage waterfall (`source · filter · build_metrics ·
+ aggregation · emit`) with source-row tables
+
+Plus a "Live debugger" entry in the existing left-nav.
+
+### 8. CLI surface
+
+The shipping CLI for this feature is **`swctl`** (or whatever the eventual
+`debugging-dsl` subcommand looks like once the CLI surface is finalized). No
+binary script is shipped in the OAP distribution for this feature.
+
+For e2e tests we keep a thin temporary helper (`test/e2e-v2/cases/.../dsl-debug.sh`
+or equivalent) so the test harness can drive the API without depending on the
+yet-to-land CLI. That helper is internal to the test tree, not packaged.
+
+Operators can also drive the API directly with `curl` against
+`/dsl-debugging/*` — the routes are stable and JSON in/out (see
+[Session API](#42-dsl-debug-session-api) and
+the walkthrough in *General usage docs* below).
+
+### 9. What is explicitly out of scope
+
+- **L2 aggregation, cross-node merge, storage round-trip, query path.** The
+ debugger stops at the L1 emit (`store`) stage. End-to-end debugging (L1 → L2 →
+ persistence → query → dashboard) is a separate problem and not part of this
+ SWIP.
+- **DSL chaining across debug sessions.** A LAL rule's `output_metric` flows into
+ MAL in production, but a debug session does not transparently follow it. The
+ operator picks one DSL per session.
+- **Behavioural changes to the data path.** Rule grammar and semantics unchanged.
+ DSL compilers' data-path behavior on a non-debug ingest is bytecode-identical
+ to today modulo one additional capture call.
+- **Storage.** Captured samples never persist; they live in the per-session ring
+ buffer until TTL.
+- **Runtime-rule hot-update.** Unchanged. The debug plugin is a sibling, not a
+ layer.
+
+### 10. Phased delivery
+
+1. **Phase 0 — `admin-server` extraction + read-only OAL management.**
+ - Pure refactor portion: stand up the `admin-server` module, migrate
+ `receiver-runtime-rule`'s embedded HTTP server onto it, delete the
+ old runtime-rule HTTP server class. Endpoints and behaviour
+ unchanged.
+ - New surface: ship the read-only OAL management endpoints
+ (`GET /runtime/oal/files`, `GET /runtime/oal/files/{name}`,
+ `GET /runtime/oal/rules`, `GET /runtime/oal/rules/{ruleName}`) on
+ `admin-server`. This is the prerequisite for the OAL debugger's rule
+ picker (Phase 3).
+ - This phase is mergeable on its own — it gives operators a runtime
+ view of OAL rules even before any DSL debugging lands — and unblocks
+ the later phases.
+2. **Phase 1 — MAL single-node.** Add `GateHolder` + `RuleKey` + `DebugRecorder` marker (server-core); add `MALDebug` + `MALDebugRecorder` (analyzer/meter-analyzer);
+ `GateHolder` + `RuleKey` in `server-core`. Stand up the
+ `dsl-debugging` module
+ with single-node `POST/GET/STOP` registered on `admin-server`. Chain
+ capture hooks in `MALMethodChainCodegen.emitChainStatements` plus
+ file-level filter and post-chain `meter_build` / `meter_emit` hooks
+ in `Analyzer.doAnalysis`. UI MAL Waterfall lights up. e2e-only helper
+ script under `test/e2e-v2/`; shipped CLI lands later via `swctl`.
+3. **Phase 2 — LAL single-node.** Block-level hooks in
+ `LALClassGenerator.generateExecuteMethod`; terminal-output capture in
+ `RecordSinkListener.parse`; metric-output capture in
+ `MetricExtractor.submitMetrics`. Statement-level capture is opt-in via
+ `granularity: "statement"` and emits `LALDebug.captureLine(...)` per
+ meaningful AST statement under a compile-time flag. UI LAL grid lights
+ up.
+4. **Phase 3 — OAL single-node.** Per-clause hooks in the OAL FreeMarker
+ template `dispatcher/doMetrics.ftl` — `source` / `filter[i]` / `build` /
+ `aggregation` / `emit`. UI OAL waterfall lights up. Picker fed by the
+ read-only OAL management API shipped in Phase 0.
+5. **Phase 4 — Cluster fan-out.** `InstallDebugSession` /
+ `CollectDebugSamples` / `StopDebugSession` / `StopByClientId` on the
+ admin-internal gRPC bus, aggregation in the receiving node, `nodes[]`
+ in the response shape.
+6. **Phase 5 — UI polish + docs.** End-to-end e2e test (push → debug
+ session → payload assertion),
+ `docs/en/setup/backend/dsl-debug-session-api.md`, screenshot
+ walkthrough.
+
+Phase 0 is independent. Phases 1-3 then deliver per-DSL value independently;
+Phase 4 unlocks cluster usage; Phase 5 lands the ops story.
+
+## Imported Dependencies libs and their licenses
+
+None. Implementation stays inside the existing module set (Javassist for codegen
+already depended on, Apache 2.0; Armeria HTTP host reused from `admin-server`,
+Apache 2.0; admin-internal gRPC bus reused from `admin-server`).
+
+## Compatibility
+
+> ### Operator-facing change — runtime-rule HTTP keys consolidate onto `admin-server`
+>
+> This SWIP introduces **one** operator-facing config break, deliberately
+> scoped to the recently-shipped runtime-rule preview (PR #13851). No
+> released telemetry/query APIs are moved or removed.
+>
+> | What changes | What operators do |
+> |-----------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------|
+> | Runtime-rule's HTTP-server keys (`receiver-runtime-rule.rest*` / `SW_RECEIVER_RUNTIME_RULE_REST_*`) are removed; their values move to `admin-server:` (`SW_ADMIN_SERVER_*`). The runtime-rule URL paths and the port number (default 17128) are unchanged. | Operators using `SW_RECEIVER_RUNTIME_RULE=default` must also set `SW_ADMIN_SERVER=default`. OAP fails fast at startup if runtime-rule is enabled without admin-server. |
+>
+> **Released APIs left untouched:**
+>
+> - `GET /status/config/ttl` stays on the query port (12800) via
+> `status-query-plugin/TTLConfigQueryHandler`.
+> - `GET /debugging/config/dump` and the other `/debugging/query/*`
+> step-through endpoints stay on the query port via
+> `status-query-plugin/DebuggingHTTPHandler`.
+> - `/runtime/rule/*` URL paths and the runtime-rule admin port (17128)
+> are preserved; only the YAML config keys feeding them change.
+
+- **Storage structure**: no change. Sessions live in memory only.
+- **Network protocols**: additive only. New `/dsl-debugging/*` HTTP
+ endpoints on the new shared `admin-server` Armeria server (default port
+ `17128`). The existing `/runtime/rule/*` endpoints migrate to the same
+ server with identical paths and behaviour — operator-visible URLs are
+ unchanged. The `status-query-plugin`'s existing query-port `/debugging/*`
+ routes (config dump, MQE step-through, trace step-through) are not moved
+ and not affected. New admin-internal RPC types `InstallDebugSession` /
+ `CollectDebugSamples` / `StopDebugSession` / `StopByClientId` — peers
+ running an older version simply do not register the handler and reply
+ `unimplemented`; the fanout records this as `install_failed` and
+ continues.
+- **Configuration — clean migration.** `admin-server` is brand new in this
+ SWIP (no prior release; nothing to be backward-compatible with). The
+ runtime-rule plugin's HTTP-server keys (`receiver-runtime-rule.restHost`,
+ `restPort`, `restContextPath`, `restIdleTimeOut`, `restAcceptQueueSize`,
+ `httpMaxRequestHeaderSize` and their `SW_RECEIVER_RUNTIME_RULE_REST_*`
+ env vars) are removed in this release; their equivalents move to the
+ new `admin-server:` block (`SW_ADMIN_SERVER_*`). The runtime-rule plugin
+ keeps only its non-HTTP config (reconciler interval, self-heal
+ threshold). Operators upgrading from the runtime-rule preview release
+ must add `SW_ADMIN_SERVER=default` (or the YAML equivalent) alongside
+ their existing `SW_RECEIVER_RUNTIME_RULE=default`. If
+ `receiver-runtime-rule.selector=default` is set without `admin-server`
+ enabled, OAP fails fast at startup with an explicit error pointing
+ operators at the new block — no silent endpoint disappearance.
+- **`docs/en/changes/changes.md`** documents the rename in the upgrade
+ notes section so the change is not surprise-discovered.
+- **DSL behavior**: bytecode-identical to today on the non-debug path.
+- **Wire breakage**: none.
+
+## General usage docs
+
+Two short walkthroughs — first the Phase-0 OAL discovery, then a Phase-1
+MAL debug session. Final version will move to
+`docs/en/setup/backend/dsl-debug-session-api.md`.
+
+```bash
+# 0. The endpoint lives on the shared admin-server port (default 17128) — the
+# same port runtime-rule already uses. admin-server must be enabled in
+# application.yml; ships disabled by default. Same security caveats as
+# runtime-rule (no auth — gateway-protect, never expose publicly).
+```
+
+**(a) Browse loaded OAL rules** (Phase 0, read-only):
+
+```bash
+# List loaded .oal files.
+curl -s http://localhost:17128/runtime/oal/files | jq .
+# → [
+# {"name":"core","path":"core.oal","ruleCount":86,"status":"LOADED"},
+# {"name":"browser","path":"browser.oal","ruleCount":14,"status":"LOADED"},
+# ...
+# ]
+
+# Drill into one file's rules.
+curl -s http://localhost:17128/runtime/oal/files/core | jq '.rules[0:3]'
+# → [
+# {"ruleName":"service_resp_time","line":2,"sourceScope":"Service",
+# "expression":"service_resp_time = from(Service.latency).longAvg();",
+# "function":"longAvg","filters":[]},
+# {"ruleName":"endpoint_cpm","line":14,"sourceScope":"Endpoint", ...},
+# ...
+# ]
+
+# Or pull the flat picker list the UI uses directly.
+curl -s http://localhost:17128/runtime/oal/rules | jq 'map(.ruleName)' | head
+```
+
+**(b) Run a DSL debug session** (Phase 1, MAL single-node):
+
+```bash
+CLIENT_ID=$(uuidgen) # one per debug context — for the CLI that's typically per shell invocation
+
+# 1. Start a session for the OTEL VM rule. catalog/name/ruleName/clientId go
+# in the query string; the optional JSON body tunes per-session limits.
+# Any prior session bound to $CLIENT_ID is auto-cleaned up before this
+# one is allocated.
+curl -s -X POST \
+ "http://localhost:17128/dsl-debugging/session?catalog=otel-rules&name=vm&ruleName=vm_cpu_total_percentage&clientId=$CLIENT_ID" \
+ -H 'Content-Type: application/json' \
+ -d '{"recordCap":1000,"retentionMillis":300000}'
+# → {"sessionId":"76b3266a-...",
+# "createdAt":1777967921000,"retentionDeadline":1777968221000,
+# "installed":{"created":3,"total":5},
+# "peers":[{"peer":"10.0.1.12:17129","ack":"INSTALLED",...}, ...],
+# "priorCleanup":{"local":{"stoppedSessionIds":["..."]},"peers":[...]}}
+
+# 2. Poll the result while traffic flows.
+curl -s http://localhost:17128/dsl-debugging/session/76b3266a-... | jq .
+# → MAL-shaped payload with steps[] and per-step samples[],
+# plus nodes[] showing per-peer capture coverage.
+
+# 3. Stop early (optional — the session also auto-expires at retention TTL,
+# or implicitly when the same clientId starts another session).
+curl -s -X POST http://localhost:17128/dsl-debugging/session/76b3266a-.../stop
+```
+
+For an OAL debug session (Phase 3), step 1 is preceded by step (a) above —
+the operator picks a `ruleName` from `/runtime/oal/rules` and posts
+`{"clientId": "...", "catalog": "oal", "name": "core", "ruleName": "endpoint_cpm", ...}`.
+
+The UI `Live debugger` view drives the same endpoints and renders the result with
+the Waterfall / grid / waterfall views from the Runtime Rule Admin design.
+
+---
+
+## Appendix — Open questions for review
+
+1. **Per-rule capture cost when idle.** Benchmarked target: ≤ 1 % overhead per stage
+ on a no-op `if (DEBUG.gate)` JIT-eliminated path. To be measured under
+ `oap-server-bench` during Phase 1.
+2. **Cap behavior.** Hitting the record cap or the capture window moves the
+ session from CAPTURING to CAPTURED. Per-field truncation does not terminate
+ the session — it just shortens the offending value with a `… +N chars
+ truncated` marker so the rest of the record remains useful. Captured samples
+ are immutable — no ring buffer, no drop-oldest, no overwrite. The payload
+ stays stable across polls until the operator stops the session or the
+ retention timeout fires. UI surfaces the terminal cause (`reason:
+ "record_cap" | "window_elapsed" | "manual_stop"`). Acceptable?
+3. **Reaffirming scope.** This proposal is parsing/transformation debugging only —
+ bounded by the L1 emit at the tail of each DSL. No L2, no cross-node merge, no
+ storage / query round-trip. Anything beyond L1 (e.g. "why did this metric
+ appear differently after L2 aggregation?") is a separate problem and stays
+ outside this SWIP. Confirm.
diff --git a/docs/en/swip/readme.md b/docs/en/swip/readme.md
index c36edba506..b0344ee83e 100644
--- a/docs/en/swip/readme.md
+++ b/docs/en/swip/readme.md
@@ -68,10 +68,12 @@ All accepted and proposed SWIPs can be found in [here](https://github.com/apache
## Known SWIPs
-Next SWIP Number: 13
+Next SWIP Number: 14
### Proposed SWIPs
+- [SWIP-13 Live Debugger for MAL / LAL / OAL](SWIP-13.md)
+
### Accepted SWIPs
- [SWIP-12 Support WeChat & Alipay Mini Program Monitoring](SWIP-12.md)
diff --git a/docs/menu.yml b/docs/menu.yml
index 94cd2e8e23..1ad605a84c 100644
--- a/docs/menu.yml
+++ b/docs/menu.yml
@@ -402,8 +402,22 @@ catalog:
catalog:
- name: "Dynamic Code Generation and Debugging"
path: "/en/operation/dynamic-code-generation-debugging"
- - name: "Runtime Rule Hot-Update"
- path: "/en/setup/backend/backend-runtime-rule-api"
+ - name: "Admin API"
+ catalog:
+ - name: "Introduction"
+ path: "/en/setup/backend/admin-api/readme"
+ - name: "Runtime Rule Hot-Update API"
+ path: "/en/setup/backend/admin-api/runtime-rule"
+ - name: "DSL Debug API"
+ catalog:
+ - name: "Overview"
+ path: "/en/setup/backend/admin-api/dsl-debugging"
+ - name: "MAL"
+ path: "/en/setup/backend/admin-api/dsl-debugging-mal"
+ - name: "OAL"
+ path: "/en/setup/backend/admin-api/dsl-debugging-oal"
+ - name: "LAL"
+ path: "/en/setup/backend/admin-api/dsl-debugging-lal"
- name: "Security Notice"
path: "/en/security/readme"
- name: "Academy"
diff --git a/oap-server/analyzer/agent-analyzer/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/trace/parser/listener/DatabaseSlowStatementBuilder.java b/oap-server/analyzer/agent-analyzer/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/trace/parser/listener/DatabaseSlowStatementBuilder.java
index 1cd2c92f96..178acb69b1 100644
--- a/oap-server/analyzer/agent-analyzer/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/trace/parser/listener/DatabaseSlowStatementBuilder.java
+++ b/oap-server/analyzer/agent-analyzer/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/trace/parser/listener/DatabaseSlowStatementBuilder.java
@@ -18,6 +18,7 @@
package org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener;
+import com.google.gson.JsonObject;
import lombok.Getter;
import lombok.Setter;
import lombok.extern.slf4j.Slf4j;
@@ -131,6 +132,22 @@ public class DatabaseSlowStatementBuilder implements LALOutputBuilder {
this.serviceName = NAMING_CONTROL.formatServiceName(serviceName);
}
+ @Override
+ public String outputToJson() {
+ final JsonObject obj = new JsonObject();
+ obj.addProperty("type", getClass().getSimpleName());
+ obj.addProperty("name", name());
+ obj.addProperty("id", id);
+ obj.addProperty("traceId", traceId);
+ obj.addProperty("serviceName", serviceName);
+ obj.addProperty("layer", layer == null ? null : layer.name());
+ obj.addProperty("statement", statement);
+ obj.addProperty("latency", latency);
+ obj.addProperty("timestamp", timestamp);
+ obj.addProperty("timeBucket", timeBucket);
+ return obj.toString();
+ }
+
public DatabaseSlowStatement toDatabaseSlowStatement() {
DatabaseSlowStatement dbSlowStat = new DatabaseSlowStatement();
dbSlowStat.setId(id);
diff --git a/oap-server/analyzer/agent-analyzer/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/trace/parser/listener/SampledTraceBuilder.java b/oap-server/analyzer/agent-analyzer/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/trace/parser/listener/SampledTraceBuilder.java
index 5c7068cb74..d67d023efc 100644
--- a/oap-server/analyzer/agent-analyzer/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/trace/parser/listener/SampledTraceBuilder.java
+++ b/oap-server/analyzer/agent-analyzer/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/trace/parser/listener/SampledTraceBuilder.java
@@ -20,6 +20,7 @@ package org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener
import com.google.common.base.Preconditions;
import com.google.common.base.Strings;
+import com.google.gson.JsonObject;
import lombok.Getter;
import lombok.Setter;
import lombok.extern.slf4j.Slf4j;
@@ -157,6 +158,26 @@ public class SampledTraceBuilder implements LALOutputBuilder {
sourceReceiver.receive(toEntity());
}
+ @Override
+ public String outputToJson() {
+ final JsonObject obj = new JsonObject();
+ obj.addProperty("type", getClass().getSimpleName());
+ obj.addProperty("name", name());
+ obj.addProperty("traceId", traceId);
+ obj.addProperty("uri", uri);
+ obj.addProperty("latency", latency);
+ obj.addProperty("reason", reason == null ? null : reason.name());
+ obj.addProperty("layer", layer);
+ obj.addProperty("serviceName", serviceName);
+ obj.addProperty("serviceInstanceName", serviceInstanceName);
+ obj.addProperty("processId", processId);
+ obj.addProperty("destProcessId", destProcessId);
+ obj.addProperty("componentId", componentId);
+ obj.addProperty("detectPoint", detectPoint == null ? null : detectPoint.name());
+ obj.addProperty("timestamp", timestamp);
+ return obj.toString();
+ }
+
public void validate() {
Preconditions.checkArgument(!Strings.isNullOrEmpty(traceId), "traceId can't be empty");
Preconditions.checkArgument(!Strings.isNullOrEmpty(uri), "uri can't be empty");
diff --git a/oap-server/analyzer/log-analyzer/CLAUDE.md b/oap-server/analyzer/log-analyzer/CLAUDE.md
index fd88f66a78..74a93836b8 100644
--- a/oap-server/analyzer/log-analyzer/CLAUDE.md
+++ b/oap-server/analyzer/log-analyzer/CLAUDE.md
@@ -374,14 +374,17 @@ oap-server/analyzer/dsl-scripts-test/src/test/resources/scripts/lal/test-lal/
## Debug Output
-When `SW_DYNAMIC_CLASS_ENGINE_DEBUG=true` environment variable is set, generated `.class` files are written to disk for inspection:
+When `SW_DYNAMIC_CLASS_ENGINE_DEBUG=true` environment variable is set, generated `.class` files are written to disk for inspection. Each `.class` is paired with a `<ClassName>.java` sidecar — the verbatim Java source the codegen fed Javassist:
```
{skywalking}/lal-rt/
*.class - Generated LalExpression .class files
+ *.java - Javassist compile input (synthetic; for IDE source-attach)
```
-This is the same env variable used by OAL. Useful for debugging code generation issues or comparing V1 vs V2 output. In tests, use `setClassOutputDir(dir)` instead.
+The `.java` sidecar exists so IDE source-attach renders the actual codegen input directly without relying on FernFlower / a decompiler. Javassist-emitted bytecode often confuses decompilers (no `goto` consolidation, slot reuse with mixed types, debug-injected `if (gate.isGateOn()) { ... }` chains), and FernFlower frequently bails to "compiled code" stubs. The source-attach path always works and shows the EXACT code Javassist compiled — gate field, probe call sites, the `_extractor()` / ` [...]
+
+When `SW_DSL_DEBUGGING_INJECTION_ENABLED=true` is also set, the codegen emits the per-rule `GateHolder debug` field plus `LALDebug.captureXxx(...)` probe sites at every block / per-statement boundary. Both `.class` and `.java` reflect the with-debug shape. The two env vars are independent: `SW_DYNAMIC_CLASS_ENGINE_DEBUG` controls disk dump; `SW_DSL_DEBUGGING_INJECTION_ENABLED` controls codegen branch. In tests, use `setClassOutputDir(dir)` instead.
## Testing Framework (server-testing module)
diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALBlockCodegen.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALBlockCodegen.java
index 5a0ae97b89..7ffe136903 100644
--- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALBlockCodegen.java
+++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALBlockCodegen.java
@@ -205,6 +205,13 @@ final class LALBlockCodegen {
LALDefCodegen.generateDefStatement(
sb, (LALScriptModel.DefStatement) stmt, genCtx);
}
+ // Statement-level captureLine — opt-in on the recorder side via
+ // session granularity. The verbatim source text is paired with the
+ // line number so the UI can render "line N: <fragment>" without a
+ // round-trip to the source file. Block-mode recorders skip; the
+ // codegen-side helper is also a no-op when injection is off.
+ LALCodegenHelper.emitCaptureLine(
+ sb, genCtx.ruleName, stmt.getSourceLine(), stmt.getSourceText());
}
}
diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALClassGenerator.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALClassGenerator.java
index 3d55c3cf61..082529d700 100644
--- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALClassGenerator.java
+++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALClassGenerator.java
@@ -36,6 +36,7 @@ import javassist.CtNewMethod;
import lombok.extern.slf4j.Slf4j;
import org.apache.skywalking.oap.log.analyzer.v2.compiler.rt.LalExpressionPackageHolder;
import org.apache.skywalking.oap.server.core.classloader.BytecodeClassDefiner;
+import org.apache.skywalking.oap.server.core.dsldebug.DSLDebugCodegenSwitch;
import org.apache.skywalking.oap.server.core.source.LogBuilder;
import org.apache.skywalking.oap.log.analyzer.v2.dsl.LalExpression;
import org.apache.skywalking.oap.server.core.WorkPath;
@@ -84,6 +85,13 @@ public final class LALClassGenerator {
private Class<?> inputType;
private Class<?> outputType;
private String yamlSource;
+ /**
+ * Optional content hash threaded into every generated rule's {@code GateHolder}
+ * constructor argument. Stamped onto every captured debug record so a UI / CLI
+ * session can detect mid-session hot-update boundaries. Defaults to the empty
+ * string when callers don't supply one.
+ */
+ private String content = "";
// ==================== Parser type detection ====================
@@ -117,6 +125,8 @@ public final class LALClassGenerator {
final Class<?> outputType;
final List<PrivateMethod> privateMethods = new ArrayList<>();
final Map<String, Integer> methodCounts = new HashMap<>();
+ /** Rule's name — emitted as the first arg of every probe call site. */
+ String ruleName = "";
// Set by generateExtraLogAccess for primitive optimization in callers.
// Reset to null by generateValueAccess at the start of each value access.
@@ -242,6 +252,16 @@ public final class LALClassGenerator {
this.yamlSource = yamlSource;
}
+ /**
+ * Sets the content hash baked into the next compiled rule's
+ * {@code GateHolder} constructor argument. Caller-supplied so the
+ * runtime-rule hot-update path can pass its already-computed
+ * {@code ContentHash} through unchanged. {@code null} → empty string.
+ */
+ public void setContent(final String content) {
+ this.content = content == null ? "" : content;
+ }
+
private String makeClassName(final String defaultPrefix) {
if (classNameHint != null) {
return dedupClassName(PACKAGE_PREFIX + buildHintedName());
@@ -300,6 +320,31 @@ public final class LALClassGenerator {
}
}
+ /**
+ * Adds the {@code public final GateHolder debug = new GateHolder("...")} instance
+ * field and the {@code debugHolder()} accessor on the generated rule class.
+ * Mandatory for every compiled rule because {@code execute()} embeds
+ * {@code if (this.debug.isGateOn()) LALDebug.captureXxx(this.debug, ...)} call sites
+ * that read the field directly.
+ */
+ private void emitDebugHolderMembers(final CtClass ctClass) throws javassist.CannotCompileException {
+ if (!DSLDebugCodegenSwitch.isInjectionEnabled()) {
+ // Injection off — fall back to the LalExpression.debugHolder() default
+ // (null), no GateHolder field, no probe call sites.
+ return;
+ }
+ final String escapedContent = LALCodegenHelper.escapeJava(content);
+ // Per-rule capture binding — instance field, lowercase per Java
+ // convention (it's a final but not a static-final constant).
+ ctClass.addField(javassist.CtField.make(
+ "public final " + LALCodegenHelper.GATE_HOLDER_FQCN + " debug = new "
+ + LALCodegenHelper.GATE_HOLDER_FQCN + "(\"" + escapedContent + "\");",
+ ctClass));
+ ctClass.addMethod(CtNewMethod.make(
+ "public " + LALCodegenHelper.GATE_HOLDER_FQCN + " debugHolder() { return this.debug; }",
+ ctClass));
+ }
+
private void writeClassFile(final CtClass ctClass) {
if (classOutputDir == null) {
return;
@@ -502,6 +547,10 @@ public final class LALClassGenerator {
"org.apache.skywalking.oap.log.analyzer.v2.dsl.LalExpression"));
ctClass.addConstructor(CtNewConstructor.defaultConstructor(ctClass));
+ // Per-rule capture binding the generated probes read via this.debug.isGateOn().
+ // Symmetric with MAL's holder member; see LALCodegenHelper.emitCaptureCall.
+ emitDebugHolderMembers(ctClass);
+
// Add private methods BEFORE execute so Javassist can resolve calls
for (final PrivateMethod pm : genCtx.privateMethods) {
final javassist.CtMethod ctMethod = CtNewMethod.make(pm.source, ctClass);
@@ -534,12 +583,65 @@ public final class LALClassGenerator {
classNameHint != null ? classNameHint : className));
writeClassFile(ctClass);
+ writeSourceFile(ctClass, genCtx, executeBody);
final Class<?> clazz = defineClass(ctClass);
ctClass.detach();
return (LalExpression) clazz.getDeclaredConstructor().newInstance();
}
+ /**
+ * Mirror of {@link #writeClassFile}: when {@code classOutputDir} is set
+ * (debug build), dump the Javassist-input Java source of the generated
+ * class as a sibling {@code <ClassName>.java} file. The {@code SourceFile}
+ * attribute on the {@code .class} points at this name, so IDEA's
+ * source-attach renders it directly without ever running FernFlower —
+ * the user sees the EXACT code that Javassist compiled, not a decompiler
+ * approximation.
+ */
+ private void writeSourceFile(final CtClass ctClass,
+ final GenCtx genCtx,
+ final String executeBody) {
+ if (classOutputDir == null) {
+ return;
+ }
+ if (!classOutputDir.exists()) {
+ classOutputDir.mkdirs();
+ }
+ final File file = new File(classOutputDir, ctClass.getSimpleName() + ".java");
+ final StringBuilder sb = new StringBuilder();
+ sb.append("// Synthetic source — Javassist compile input for ")
+ .append(ctClass.getSimpleName()).append("\n")
+ .append("// Written when SW_DYNAMIC_CLASS_ENGINE_DEBUG is on; used by IDE\n")
+ .append("// source-attach to render the bytecode without FernFlower.\n\n");
+ sb.append("package ").append(ctClass.getPackageName()).append(";\n\n");
+ sb.append("public class ").append(ctClass.getSimpleName())
+ .append(" implements ")
+ .append(LalExpression.class.getName())
+ .append(" {\n\n");
+ if (DSLDebugCodegenSwitch.isInjectionEnabled()) {
+ // Same escapedContent the bytecode emits — the verbatim LAL DSL
+ // for this rule, escaped for a Java string literal so the
+ // sidecar source compiles structurally identical to the .class.
+ sb.append(" public final ").append(LALCodegenHelper.GATE_HOLDER_FQCN)
+ .append(" debug = new ").append(LALCodegenHelper.GATE_HOLDER_FQCN)
+ .append("(\"").append(LALCodegenHelper.escapeJava(content))
+ .append("\");\n\n");
+ sb.append(" public ").append(LALCodegenHelper.GATE_HOLDER_FQCN)
+ .append(" debugHolder() { return this.debug; }\n\n");
+ }
+ for (final PrivateMethod pm : genCtx.privateMethods) {
+ sb.append(" ").append(pm.source.replace("\n", "\n ")).append("\n\n");
+ }
+ sb.append(" ").append(executeBody.replace("\n", "\n ")).append("\n");
+ sb.append("}\n");
+ try (java.io.FileWriter w = new java.io.FileWriter(file)) {
+ w.write(sb.toString());
+ } catch (Exception e) {
+ log.warn("Failed to write source file {}: {}", file, e.getMessage());
+ }
+ }
+
/**
* Loads a generated class through the configured {@link #targetClassLoader} when set
* (runtime-rule hot-update path: class lands in the per-file {@code RuleClassLoader}),
@@ -591,6 +693,7 @@ public final class LALClassGenerator {
private String generateExecuteMethod(final LALScriptModel model,
final GenCtx genCtx) {
genCtx.resetProtoVars();
+ genCtx.ruleName = classNameHint == null ? "lal" : classNameHint;
// Generate body first so proto var declarations are collected
final StringBuilder bodyContent = new StringBuilder();
@@ -603,9 +706,28 @@ public final class LALClassGenerator {
.append(" filterSpec, ").append(EXEC_CTX).append(" ctx) {\n");
sb.append(" ").append(H).append(" h = new ").append(H).append("(ctx);\n");
- // Create the output object and store in ctx before extractor runs
+ // Create the output object and store in ctx before extractor runs.
+ // Then bind the typed input + metadata onto the builder eagerly so
+ // every downstream probe (extractor / per-statement / sink) sees a
+ // builder whose state already reflects the input — the merged
+ // tags[] view in outputToJson() is built from logData + lalTags,
+ // and would otherwise show only lalTags at the function probes
+ // since RecordSinkListener.parse() doesn't fire init() until sink
+ // time.
sb.append(" h.ctx().setOutput(new ")
.append(genCtx.outputType.getName()).append("());\n");
+ sb.append(" h.ctx().outputAsBuilder().bindInput(h.ctx().metadata(), h.ctx().input());\n");
+
+ // Push the rule's debug holder onto ctx so downstream analyzer wrappers
+ // (RecordSinkListener.parse, MetricExtractor.submitMetrics) fire their
+ // terminal-output probes without re-resolving the rule. Skipped entirely
+ // when injection is off — there's no debug field to read.
+ if (DSLDebugCodegenSwitch.isInjectionEnabled()) {
+ sb.append(" ctx.setDebugHolder(this.debug);\n");
+ }
+
+ // Top-of-pipeline capture: raw log body view as the rule first sees it.
+ LALCodegenHelper.emitCaptureCall(sb, "Text", genCtx.ruleName, 0, "ctx", "");
// Insert _p + proto var declarations if any proto field access was used
if (genCtx.usedProtoAccess) {
@@ -639,15 +761,19 @@ public final class LALClassGenerator {
} else {
sb.append(" filterSpec.text(ctx);\n");
}
+ LALCodegenHelper.emitCaptureCall(sb, "Parser", genCtx.ruleName, 0, "ctx", "");
} else if (stmt instanceof LALScriptModel.JsonParser) {
sb.append(" filterSpec.json(ctx);\n");
+ LALCodegenHelper.emitCaptureCall(sb, "Parser", genCtx.ruleName, 0, "ctx", "");
} else if (stmt instanceof LALScriptModel.YamlParser) {
sb.append(" filterSpec.yaml(ctx);\n");
+ LALCodegenHelper.emitCaptureCall(sb, "Parser", genCtx.ruleName, 0, "ctx", "");
} else if (stmt instanceof LALScriptModel.AbortStatement) {
sb.append(" filterSpec.abort(ctx);\n");
} else if (stmt instanceof LALScriptModel.ExtractorBlock) {
LALBlockCodegen.generateExtractorMethod(
sb, (LALScriptModel.ExtractorBlock) stmt, genCtx);
+ LALCodegenHelper.emitCaptureCall(sb, "Extractor", genCtx.ruleName, 0, "ctx", "");
} else if (stmt instanceof LALScriptModel.SinkBlock) {
final LALScriptModel.SinkBlock sink = (LALScriptModel.SinkBlock) stmt;
if (sink.getStatements().isEmpty()) {
@@ -655,6 +781,10 @@ public final class LALClassGenerator {
} else {
LALBlockCodegen.generateSinkMethod(sb, sink, genCtx);
}
+ // No probe at the sink boundary itself — the terminal capture happens
+ // downstream in RecordSinkListener.parse (output record) and
+ // MetricExtractor.submitMetrics (output metric), and only fires for
+ // records the sink kept. SWIP-13: A7.
} else if (stmt instanceof LALScriptModel.IfBlock) {
generateTopLevelIfBlock(sb, (LALScriptModel.IfBlock) stmt, genCtx);
} else if (stmt instanceof LALScriptModel.DefStatement) {
diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALCodegenHelper.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALCodegenHelper.java
index a463d13b4e..1da2531613 100644
--- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALCodegenHelper.java
+++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALCodegenHelper.java
@@ -23,6 +23,7 @@ import java.util.List;
import java.util.Map;
import java.util.Set;
import org.apache.skywalking.apm.network.logging.v3.LogData;
+import org.apache.skywalking.oap.server.core.dsldebug.DSLDebugCodegenSwitch;
/**
* Static utility constants and methods extracted from {@link LALClassGenerator}
@@ -66,10 +67,65 @@ final class LALCodegenHelper {
}
+ /** FQCN of the per-rule capture binding the generated rule class carries. */
+ static final String GATE_HOLDER_FQCN =
+ "org.apache.skywalking.oap.server.core.dsldebug.GateHolder";
+
+ /** FQCN of the LAL probe class generated bytecode forwards to. */
+ static final String LAL_DEBUG_FQCN =
+ "org.apache.skywalking.oap.log.analyzer.v2.dsldebug.LALDebug";
+
private LALCodegenHelper() {
// utility class
}
+ /**
+ * Emits a gate-guarded {@code LALDebug.captureXxx(...)} call site at the current position
+ * in a generated method body. Same shape as the MAL helper — one place owns the gate-guard
+ * idiom + Java-string escaping so every probe insertion site stays uniform.
+ *
+ * @param sb buffer the emitted source line is appended to
+ * @param verb capture verb the recorder method maps to ("Text", "Parser",
+ * "Extractor", "Sink", "Line")
+ * @param ruleName rule's name — Java-string-escaped before emission
+ * @param sourceLine 1-based source-line tag (or 0 for synthetic stages)
+ * @param ctxRef the local-variable reference holding the {@code ExecutionContext}
+ * (e.g. {@code "ctx"} or {@code "h.ctx()"})
+ * @param extraArgs optional comma-prefixed string of extra arguments after ctx (e.g.
+ * {@code ", h.lastSinkBranch(), !h.lastSinkDropped()"} for sink); pass
+ * the empty string when not needed
+ */
+ static void emitCaptureCall(final StringBuilder sb, final String verb, final String ruleName,
+ final int sourceLine, final String ctxRef, final String extraArgs) {
+ if (!DSLDebugCodegenSwitch.isInjectionEnabled()) {
+ return;
+ }
+ sb.append(" if (this.debug.isGateOn()) ").append(LAL_DEBUG_FQCN)
+ .append(".capture").append(verb).append("(this.debug, \"")
+ .append(escapeJava(ruleName)).append("\", ")
+ .append(sourceLine).append(", ").append(ctxRef)
+ .append(extraArgs == null ? "" : extraArgs)
+ .append(");\n");
+ }
+
+ /**
+ * Specialised emit for {@code LALDebug.captureLine}. Same gate-guard idiom
+ * as {@link #emitCaptureCall} but threads the verbatim statement source
+ * text into the call so the captured record carries
+ * "line {N}: {sourceText}" without an out-of-band lookup.
+ */
+ static void emitCaptureLine(final StringBuilder sb, final String ruleName,
+ final int sourceLine, final String sourceText) {
+ if (!DSLDebugCodegenSwitch.isInjectionEnabled()) {
+ return;
+ }
+ sb.append(" if (this.debug.isGateOn()) ").append(LAL_DEBUG_FQCN)
+ .append(".captureLine(this.debug, \"").append(escapeJava(ruleName))
+ .append("\", ").append(sourceLine).append(", \"")
+ .append(escapeJava(sourceText == null ? "" : sourceText))
+ .append("\", h.ctx());\n");
+ }
+
static String escapeJava(final String s) {
return s.replace("\\", "\\\\")
.replace("\"", "\\\"")
diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALScriptModel.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALScriptModel.java
index e2416a49cf..e0dfe266a1 100644
--- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALScriptModel.java
+++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALScriptModel.java
@@ -46,6 +46,27 @@ public final class LALScriptModel {
// ==================== Filter statements ====================
public interface FilterStatement {
+ /**
+ * 1-based source line number of this AST statement, or {@code 0} when
+ * the statement was synthesised (e.g. a default sink). Threaded from
+ * {@code ctx.getStart().getLine()} at parse time.
+ */
+ default int getSourceLine() {
+ return 0;
+ }
+
+ /**
+ * Verbatim source text of the original DSL statement
+ * (e.g. {@code "tag 'status.code': parsed.code"}). The dsl-debugging
+ * codegen pairs this with {@link #getSourceLine()} on every
+ * statement-level capture so the UI can render
+ * "line N: {sourceText}" without round-tripping back to the source
+ * file. Returns the empty string when the statement carries no
+ * source-text origin (synthesised statements).
+ */
+ default String getSourceText() {
+ return "";
+ }
}
// ==================== Parser blocks ====================
@@ -89,6 +110,8 @@ public final class LALScriptModel {
private final String varName;
private final ValueAccess initializer;
private final String castType;
+ private int sourceLine;
+ private String sourceText = "";
public DefStatement(final String varName,
final ValueAccess initializer,
@@ -97,6 +120,12 @@ public final class LALScriptModel {
this.initializer = initializer;
this.castType = castType;
}
+
+ public DefStatement withSource(final int line, final String text) {
+ this.sourceLine = line;
+ this.sourceText = text == null ? "" : text;
+ return this;
+ }
}
// ==================== Extractor block ====================
@@ -119,6 +148,8 @@ public final class LALScriptModel {
private final ValueAccess value;
private final String castType;
private final String formatPattern;
+ private int sourceLine;
+ private String sourceText = "";
public FieldAssignment(final FieldType fieldType,
final ValueAccess value,
@@ -129,6 +160,12 @@ public final class LALScriptModel {
this.castType = castType;
this.formatPattern = formatPattern;
}
+
+ public FieldAssignment withSource(final int line, final String text) {
+ this.sourceLine = line;
+ this.sourceText = text == null ? "" : text;
+ return this;
+ }
}
@Getter
@@ -136,6 +173,8 @@ public final class LALScriptModel {
private final String fieldName;
private final ValueAccess value;
private final String castType;
+ private int sourceLine;
+ private String sourceText = "";
public OutputFieldAssignment(final String fieldName,
final ValueAccess value,
@@ -144,6 +183,12 @@ public final class LALScriptModel {
this.value = value;
this.castType = castType;
}
+
+ public OutputFieldAssignment withSource(final int line, final String text) {
+ this.sourceLine = line;
+ this.sourceText = text == null ? "" : text;
+ return this;
+ }
}
public enum FieldType {
@@ -154,10 +199,18 @@ public final class LALScriptModel {
@Getter
public static final class TagAssignment implements ExtractorStatement, FilterStatement {
private final Map<String, TagValue> tags;
+ private int sourceLine;
+ private String sourceText = "";
public TagAssignment(final Map<String, TagValue> tags) {
this.tags = Collections.unmodifiableMap(tags);
}
+
+ public TagAssignment withSource(final int line, final String text) {
+ this.sourceLine = line;
+ this.sourceText = text == null ? "" : text;
+ return this;
+ }
}
@Getter
@@ -179,6 +232,8 @@ public final class LALScriptModel {
private final Map<String, TagValue> labels;
private final ValueAccess value;
private final String valueCast;
+ private int sourceLine;
+ private String sourceText = "";
public MetricsBlock(final String name,
final ValueAccess timestampValue,
@@ -193,6 +248,12 @@ public final class LALScriptModel {
this.value = value;
this.valueCast = valueCast;
}
+
+ public MetricsBlock withSource(final int line, final String text) {
+ this.sourceLine = line;
+ this.sourceText = text == null ? "" : text;
+ return this;
+ }
}
// ==================== Sink block ====================
diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALScriptParser.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALScriptParser.java
index 9cd3f27f51..38c8942337 100644
--- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALScriptParser.java
+++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALScriptParser.java
@@ -27,6 +27,7 @@ import org.antlr.v4.runtime.CommonTokenStream;
import org.antlr.v4.runtime.ParserRuleContext;
import org.antlr.v4.runtime.RecognitionException;
import org.antlr.v4.runtime.Recognizer;
+import org.antlr.v4.runtime.misc.Interval;
import org.antlr.v4.runtime.tree.ParseTree;
import org.antlr.v4.runtime.tree.TerminalNode;
import org.apache.skywalking.lal.rt.grammar.LALLexer;
@@ -201,28 +202,35 @@ public final class LALScriptParser {
private static ExtractorStatement visitExtractorStatement(
final LALParser.ExtractorStatementContext ctx) {
+ // Capture (line, raw text) once at the statement boundary so every
+ // dispatch below — through visitFieldAssignment / visitTagStatement /
+ // ... — can stamp the resulting AST node with the same source-origin
+ // pair. dsl-debugging's per-line probe reads these back out at codegen.
+ final int sourceLine = ctx.getStart().getLine();
+ final String sourceText = rawTextOf(ctx);
+
if (ctx.defStatement() != null) {
- return (ExtractorStatement) visitDefStatement(ctx.defStatement());
+ return visitDefStatement(ctx.defStatement()).withSource(sourceLine, sourceText);
}
if (ctx.serviceStatement() != null) {
return visitFieldAssignment(FieldType.SERVICE, ctx.serviceStatement().valueAccess(),
- ctx.serviceStatement().typeCast());
+ ctx.serviceStatement().typeCast()).withSource(sourceLine, sourceText);
}
if (ctx.instanceStatement() != null) {
return visitFieldAssignment(FieldType.INSTANCE, ctx.instanceStatement().valueAccess(),
- ctx.instanceStatement().typeCast());
+ ctx.instanceStatement().typeCast()).withSource(sourceLine, sourceText);
}
if (ctx.endpointStatement() != null) {
return visitFieldAssignment(FieldType.ENDPOINT, ctx.endpointStatement().valueAccess(),
- ctx.endpointStatement().typeCast());
+ ctx.endpointStatement().typeCast()).withSource(sourceLine, sourceText);
}
if (ctx.layerStatement() != null) {
return visitFieldAssignment(FieldType.LAYER, ctx.layerStatement().valueAccess(),
- ctx.layerStatement().typeCast());
+ ctx.layerStatement().typeCast()).withSource(sourceLine, sourceText);
}
if (ctx.traceIdStatement() != null) {
return visitFieldAssignment(FieldType.TRACE_ID, ctx.traceIdStatement().valueAccess(),
- ctx.traceIdStatement().typeCast());
+ ctx.traceIdStatement().typeCast()).withSource(sourceLine, sourceText);
}
if (ctx.timestampStatement() != null) {
final ValueAccess va = visitValueAccess(ctx.timestampStatement().valueAccess());
@@ -231,21 +239,38 @@ public final class LALScriptParser {
if (ctx.timestampStatement().STRING() != null) {
format = stripQuotes(ctx.timestampStatement().STRING().getText());
}
- return new FieldAssignment(FieldType.TIMESTAMP, va, cast, format);
+ return new FieldAssignment(FieldType.TIMESTAMP, va, cast, format)
+ .withSource(sourceLine, sourceText);
}
if (ctx.tagStatement() != null) {
- return visitTagStatement(ctx.tagStatement());
+ return visitTagStatement(ctx.tagStatement()).withSource(sourceLine, sourceText);
}
if (ctx.metricsBlock() != null) {
- return visitMetricsBlock(ctx.metricsBlock());
+ return visitMetricsBlock(ctx.metricsBlock()).withSource(sourceLine, sourceText);
}
if (ctx.outputFieldStatement() != null) {
- return visitOutputFieldStatement(ctx.outputFieldStatement());
+ return visitOutputFieldStatement(ctx.outputFieldStatement())
+ .withSource(sourceLine, sourceText);
}
- // ifStatement
+ // ifStatement carries its own per-branch line info; surface the if-line
+ // here for the surrounding statement-level capture.
return (ExtractorStatement) visitIfStatement(ctx.ifStatement());
}
+ /**
+ * Verbatim source text of an ANTLR rule context — pulls from the original
+ * input stream rather than re-concatenating tokens, so whitespace inside
+ * the statement is preserved (matters for the dsl-debugging UI rendering
+ * {@code "line N: <verbatim>"}).
+ */
+ private static String rawTextOf(final ParserRuleContext ctx) {
+ if (ctx.getStart() == null || ctx.getStop() == null) {
+ return "";
+ }
+ return ctx.getStart().getInputStream().getText(
+ Interval.of(ctx.getStart().getStartIndex(), ctx.getStop().getStopIndex()));
+ }
+
private static FieldAssignment visitFieldAssignment(
final FieldType type,
final LALParser.ValueAccessContext vaCtx,
diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/DSL.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/DSL.java
index ad850c216a..0993bc6a0a 100644
--- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/DSL.java
+++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/DSL.java
@@ -17,13 +17,16 @@
package org.apache.skywalking.oap.log.analyzer.v2.dsl;
+import java.util.LinkedHashMap;
import javassist.ClassPool;
import lombok.AccessLevel;
+import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.apache.skywalking.oap.log.analyzer.v2.compiler.LALClassGenerator;
import org.apache.skywalking.oap.log.analyzer.v2.dsl.spec.filter.FilterSpec;
import org.apache.skywalking.oap.log.analyzer.v2.provider.LogAnalyzerModuleConfig;
+import org.apache.skywalking.oap.server.core.dsldebug.GateHolder;
import org.apache.skywalking.oap.server.core.source.LogMetadata;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
import org.apache.skywalking.oap.server.library.module.ModuleStartException;
@@ -41,6 +44,7 @@ import org.apache.skywalking.oap.server.library.module.ModuleStartException;
@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
public class DSL {
private final String ruleName;
+ @Getter
private final LalExpression expression;
private final FilterSpec filterSpec;
@@ -96,7 +100,29 @@ public class DSL {
generator.setOutputType(outputType);
generator.setClassNameHint(ruleName);
generator.setYamlSource(yamlSource);
+ // Pass the verbatim DSL text as the holder's "content" — dsl-debugging
+ // captures stamp this on every record so the UI renders the rule source
+ // inline. SHA-256 hash isn't useful to operators; raw text is.
+ generator.setContent(dsl);
final LalExpression expression = generator.compile(dsl);
+ // Stamp the structured rule metadata onto the per-rule GateHolder so
+ // dsl-debugging records render {ruleName, layer, outputClass} alongside
+ // the verbatim DSL. Only effective when codegen injection is enabled
+ // (otherwise debugHolder() returns null on the compiled expression).
+ final GateHolder holder = expression.debugHolder();
+ if (holder != null) {
+ final LinkedHashMap<String, String> meta = new LinkedHashMap<>();
+ if (ruleName != null && !ruleName.isEmpty()) {
+ meta.put("ruleName", ruleName);
+ }
+ if (outputType != null) {
+ meta.put("outputClass", outputType.getName());
+ }
+ if (inputType != null) {
+ meta.put("inputClass", inputType.getName());
+ }
+ holder.setMetadata(meta);
+ }
final FilterSpec filterSpec = new FilterSpec(moduleManager, config);
return new DSL(ruleName, expression, filterSpec);
} catch (Exception e) {
diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/ExecutionContext.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/ExecutionContext.java
index 2039bae36b..c47daf3f8e 100644
--- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/ExecutionContext.java
+++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/ExecutionContext.java
@@ -17,13 +17,19 @@
package org.apache.skywalking.oap.log.analyzer.v2.dsl;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonObject;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.regex.Matcher;
import lombok.Getter;
+import org.apache.skywalking.apm.network.logging.v3.LogData;
+import org.apache.skywalking.oap.log.analyzer.v2.dsldebug.LalPayloadDebugDump;
+import org.apache.skywalking.oap.log.analyzer.v2.spi.LALSourceTypeProvider;
import org.apache.skywalking.oap.meter.analyzer.v2.dsl.SampleFamily;
+import org.apache.skywalking.oap.server.core.dsldebug.GateHolder;
import org.apache.skywalking.oap.server.core.source.LALOutputBuilder;
import org.apache.skywalking.oap.server.core.source.LogMetadata;
@@ -54,6 +60,35 @@ public class ExecutionContext {
private final Map<String, Object> properties = new HashMap<>();
+ /**
+ * The rule's debug capture binding, set on entry to the generated
+ * {@code execute()} method. Downstream analyzer wrappers
+ * (RecordSinkListener, MetricExtractor) read it here to fire the
+ * terminal-output probes without having to re-resolve the rule's
+ * compiled {@code LalExpression}.
+ */
+ @Getter
+ @lombok.Setter
+ private GateHolder debugHolder;
+
+ /**
+ * Resolved {@link LALSourceTypeProvider} for this log's layer. Set
+ * by {@link org.apache.skywalking.oap.log.analyzer.v2.provider.log.listener.LogFilterListener}
+ * at parse time using {@code metadata.getLayer()}; remains
+ * {@code null} for the default agent path (no provider registered =
+ * input is {@code LogData} / {@code LogData.Builder}).
+ *
+ * <p>The provider supplies layer-level type resolution
+ * ({@link LALSourceTypeProvider#inputType()} /
+ * {@link LALSourceTypeProvider#outputType()}) at compile time. The
+ * dsl-debugging dump path renders payloads via the
+ * {@link LALOutputBuilder} the rule installed on this context; the
+ * provider is not consulted at dump time.
+ */
+ @Getter
+ @lombok.Setter
+ private LALSourceTypeProvider sourceTypeProvider;
+
public ExecutionContext() {
setProperty(KEY_PARSED, new Parsed());
}
@@ -170,6 +205,108 @@ public class ExecutionContext {
return (LALOutputBuilder) getProperty(KEY_OUTPUT);
}
+ /**
+ * Payload for the input-type sample fired at the top of the LAL
+ * pipeline (one per execution). Renders the raw input directly via
+ * the framework's native {@link LogData.Builder} renderer — does
+ * NOT consult the output {@link LALOutputBuilder}, which has just
+ * been constructed by the generated {@code execute()} body and is
+ * not yet populated by {@code init()} (init runs at the sink stage).
+ *
+ * <p>Mirrors the {@code aborted} / {@code hasParsed} flags and the
+ * parsed-map key set so operators can see which extractor branches
+ * a log walked through. No {@code output} key — that data evolves
+ * in subsequent samples and would be empty here.
+ */
+ public String inputPayloadJson() {
+ final JsonObject obj = new JsonObject();
+ obj.addProperty("aborted", shouldAbort());
+ obj.addProperty("hasParsed", parsed() != null);
+ obj.add("input", LalPayloadDebugDump.parse(renderRawInput()));
+ obj.add("parsedKeys", parsedKeysJson());
+ return obj.toString();
+ }
+
+ /**
+ * Payload for every sample after the input probe — block / statement
+ * / function / output. Renders only the {@link LALOutputBuilder}'s
+ * accumulated state (the DB-bound fields the rule has set so far,
+ * including {@code content} / {@code contentType} from the cached
+ * input body — see {@code LogBuilder.appendBodyContent}). The raw
+ * input is captured once at the top of the pipeline; repeating it
+ * on every probe would be redundant.
+ */
+ public String outputPayloadJson() {
+ final JsonObject obj = new JsonObject();
+ obj.addProperty("aborted", shouldAbort());
+ obj.addProperty("hasParsed", parsed() != null);
+ obj.add("output", LalPayloadDebugDump.parse(renderOutput()));
+ obj.add("parsedKeys", parsedKeysJson());
+ return obj.toString();
+ }
+
+ /**
+ * Native renderer for {@code ctx.input()} — used at the input probe
+ * before any builder has been initialised. Agent path delivers a
+ * {@link LogData.Builder}; receiver-typed paths (envoy ALS,
+ * arbitrary protobufs) deliver a {@link com.google.protobuf.Message}
+ * subclass. Falls back to the unknown-type hint if neither fits, so
+ * a misconfigured layer doesn't crash the dump.
+ */
+ private String renderRawInput() {
+ final Object value = input();
+ if (value == null) {
+ return "null";
+ }
+ if (value instanceof LogData.Builder) {
+ return LalPayloadDebugDump.toJson((LogData.Builder) value);
+ }
+ if (value instanceof LogData) {
+ return LalPayloadDebugDump.toJson((LogData) value);
+ }
+ if (value instanceof com.google.protobuf.Message) {
+ return LalPayloadDebugDump.toJson((com.google.protobuf.Message) value);
+ }
+ return LalPayloadDebugDump.unknownTypeHint(value);
+ }
+
+ /**
+ * Render the output payload — every LAL rule installs a
+ * {@link LALOutputBuilder} on the context, and the builder owns its
+ * own debug serialization via
+ * {@link LALOutputBuilder#outputToJson()}. The interface contract
+ * makes the dispatch a single static call.
+ *
+ * <p>If the output slot is empty (the rule aborted before
+ * installing a builder), nothing meaningful exists to dump — the
+ * helper returns a hint object so the operator can see the type
+ * the framework received and add an {@link LALOutputBuilder} for
+ * it.
+ */
+ private String renderOutput() {
+ final Object value = output();
+ if (value == null) {
+ return "null";
+ }
+ if (value instanceof LALOutputBuilder) {
+ return ((LALOutputBuilder) value).outputToJson();
+ }
+ final JsonObject hint = new JsonObject();
+ hint.addProperty("type", value.getClass().getName());
+ hint.addProperty("note", "cannot-serialize-output-no-LALOutputBuilder-impl");
+ return hint.toString();
+ }
+
+ private JsonArray parsedKeysJson() {
+ final JsonArray keys = new JsonArray();
+ if (parsed() != null && parsed().getMap() != null) {
+ for (final String key : parsed().getMap().keySet()) {
+ keys.add(key);
+ }
+ }
+ return keys;
+ }
+
public static class Parsed {
@Getter
private Matcher matcher;
diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/LalExpression.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/LalExpression.java
index 1d77c5622a..a0fd8b997d 100644
--- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/LalExpression.java
+++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/LalExpression.java
@@ -19,17 +19,34 @@
package org.apache.skywalking.oap.log.analyzer.v2.dsl;
import org.apache.skywalking.oap.log.analyzer.v2.dsl.spec.filter.FilterSpec;
+import org.apache.skywalking.oap.server.core.dsldebug.GateHolder;
/**
- * Functional interface implemented by each compiled LAL class.
+ * Implemented by each compiled LAL class.
*
* <p>Generated at startup by
* {@link org.apache.skywalking.oap.log.analyzer.v2.compiler.LALClassGenerator}
* via ANTLR4 parsing and Javassist bytecode generation.
* The generated {@code execute} method calls {@link FilterSpec} methods
* (json/text/yaml, extractor, sink) in the order defined by the LAL script.
+ *
+ * <p>{@link #debugHolder()} exposes the per-rule capture binding the
+ * generated bytecode reads on every probe call site. The interface stays
+ * single-method-callable from the analyzer's perspective ({@code execute})
+ * — {@code debugHolder} is a getter, not a workflow method.
*/
-@FunctionalInterface
public interface LalExpression {
void execute(FilterSpec filterSpec, ExecutionContext ctx);
+
+ /**
+ * Returns the per-rule capture binding for this compiled LAL rule.
+ * Codegen emits the {@code public final GateHolder debug} field + this
+ * override only when
+ * {@link org.apache.skywalking.oap.server.core.dsldebug.DSLDebugCodegenSwitch#isInjectionEnabled()
+ * dsl-debugging is enabled at boot}. When injection is off, this default
+ * returns {@code null}.
+ */
+ default GateHolder debugHolder() {
+ return null;
+ }
}
diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/spec/extractor/MetricExtractor.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/spec/extractor/MetricExtractor.java
index 2ff07fff85..49b0b0fe8b 100644
--- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/spec/extractor/MetricExtractor.java
+++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/spec/extractor/MetricExtractor.java
@@ -27,7 +27,9 @@ import java.util.stream.Collectors;
import lombok.experimental.Delegate;
import org.apache.skywalking.oap.log.analyzer.v2.dsl.ExecutionContext;
import org.apache.skywalking.oap.log.analyzer.v2.dsl.spec.AbstractSpec;
+import org.apache.skywalking.oap.log.analyzer.v2.dsldebug.LALDebug;
import org.apache.skywalking.oap.log.analyzer.v2.module.LogAnalyzerModule;
+import org.apache.skywalking.oap.server.core.dsldebug.GateHolder;
import org.apache.skywalking.oap.log.analyzer.v2.provider.LogAnalyzerModuleConfig;
import org.apache.skywalking.oap.log.analyzer.v2.provider.LogAnalyzerModuleProvider;
import org.apache.skywalking.oap.meter.analyzer.v2.dsl.Sample;
@@ -76,6 +78,14 @@ public class MetricExtractor extends AbstractSpec {
final Sample sample = builder.build();
final SampleFamily sampleFamily = SampleFamilyBuilder.newBuilder(sample).build();
+ // Terminal metric-output probe: capture the family right at the LAL→MAL hand-off.
+ // The debugger does NOT chain into the MAL pipeline that consumes this family —
+ // that's a separate session boundary.
+ final GateHolder debugHolder = ctx.getDebugHolder();
+ if (debugHolder != null && debugHolder.isGateOn()) {
+ LALDebug.captureOutputMetric(debugHolder, "", 0, ctx, sampleFamily);
+ }
+
final Optional<List<SampleFamily>> possibleMetricsContainer = ctx.metricsContainer();
if (possibleMetricsContainer.isPresent()) {
diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsldebug/LALDebug.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsldebug/LALDebug.java
new file mode 100644
index 0000000000..5a3800341b
--- /dev/null
+++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsldebug/LALDebug.java
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.log.analyzer.v2.dsldebug;
+
+import org.apache.skywalking.oap.log.analyzer.v2.dsl.ExecutionContext;
+import org.apache.skywalking.oap.meter.analyzer.v2.dsl.SampleFamily;
+import org.apache.skywalking.oap.server.core.dsldebug.DebugRecorder;
+import org.apache.skywalking.oap.server.core.dsldebug.GateHolder;
+
+/**
+ * Probe surface generated LAL bytecode calls into. Each method takes the
+ * per-rule {@link GateHolder} as its first argument and fans the typed
+ * payload to every recorder bound on that holder.
+ *
+ * <p>Symmetric with {@code MALDebug} / {@code OALDebug}. The codegen-emitted
+ * call sites all use the {@code if (this.debug.isGateOn())} idiom (see
+ * {@code LALCodegenHelper.emitCaptureCall}).
+ */
+public final class LALDebug {
+
+ private LALDebug() {
+ }
+
+ /** Functional dispatcher used by every probe to forward to one recorder. */
+ @FunctionalInterface
+ private interface Dispatch {
+ void to(LALDebugRecorder recorder);
+ }
+
+ public static void captureText(final GateHolder holder, final String rule,
+ final int sourceLine, final ExecutionContext ctx) {
+ fanOut(holder, r -> r.appendText(rule, sourceLine, ctx));
+ }
+
+ public static void captureParser(final GateHolder holder, final String rule,
+ final int sourceLine, final ExecutionContext ctx) {
+ fanOut(holder, r -> r.appendParser(rule, sourceLine, ctx));
+ }
+
+ public static void captureExtractor(final GateHolder holder, final String rule,
+ final int sourceLine, final ExecutionContext ctx) {
+ fanOut(holder, r -> r.appendExtractor(rule, sourceLine, ctx));
+ }
+
+ public static void captureLine(final GateHolder holder, final String rule,
+ final int sourceLine, final String sourceText,
+ final ExecutionContext ctx) {
+ fanOut(holder, r -> r.appendLine(rule, sourceLine, sourceText, ctx));
+ }
+
+ public static void captureOutputRecord(final GateHolder holder, final String rule,
+ final int sourceLine, final ExecutionContext ctx,
+ final String outputClass) {
+ fanOut(holder, r -> r.appendOutputRecord(rule, sourceLine, ctx, outputClass));
+ }
+
+ public static void captureOutputMetric(final GateHolder holder, final String rule,
+ final int sourceLine, final ExecutionContext ctx,
+ final SampleFamily family) {
+ fanOut(holder, r -> r.appendOutputMetric(rule, sourceLine, ctx, family));
+ }
+
+ private static void fanOut(final GateHolder holder, final Dispatch dispatch) {
+ final DebugRecorder[] snapshot = holder.getRecorders();
+ for (int i = 0; i < snapshot.length; i++) {
+ final DebugRecorder recorder = snapshot[i];
+ if (recorder.isCaptured()) {
+ continue;
+ }
+ dispatch.to((LALDebugRecorder) recorder);
+ }
+ }
+}
diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsldebug/LALDebugRecorder.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsldebug/LALDebugRecorder.java
new file mode 100644
index 0000000000..ee30d42bad
--- /dev/null
+++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsldebug/LALDebugRecorder.java
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.log.analyzer.v2.dsldebug;
+
+import org.apache.skywalking.oap.log.analyzer.v2.dsl.ExecutionContext;
+import org.apache.skywalking.oap.meter.analyzer.v2.dsl.SampleFamily;
+import org.apache.skywalking.oap.server.core.dsldebug.DebugRecorder;
+
+/**
+ * Per-session recorder for LAL rules.
+ *
+ * <p>LAL keeps a {@code sourceLine} on every probe — unlike MAL/OAL, the LAL
+ * grammar maps each AST statement to a clean source line, and the
+ * statement-level granularity option needs that line so the UI can highlight
+ * the originating statement.
+ *
+ * <p>Stages:
+ * <ul>
+ * <li>{@code text} — raw log body view as the rule first sees it.</li>
+ * <li>{@code parser} — after {@code json}/{@code yaml}/{@code text} populates
+ * {@code ctx.parsed()}.</li>
+ * <li>{@code extractor} — after the {@code extractor} block runs (typed builder mid-flight).</li>
+ * <li>{@code line} — per-statement capture (opt-in via {@code granularity: "statement"}).</li>
+ * <li>{@code outputRecord} — terminal: typed output builder before {@code complete()},
+ * captured iff the record survived the sink.</li>
+ * <li>{@code outputMetric} — terminal: {@link SampleFamily} handed off to MAL,
+ * captured iff the metric extractor produced one.</li>
+ * </ul>
+ *
+ * <p>The {@code sink} stage is folded into {@code outputRecord} / {@code outputMetric}
+ * — sink either keeps the record (an output appears) or drops it (no output capture).
+ * No separate {@code sink} verb needed.
+ */
+public interface LALDebugRecorder extends DebugRecorder {
+
+ void appendText(String rule, int sourceLine, ExecutionContext ctx);
+
+ void appendParser(String rule, int sourceLine, ExecutionContext ctx);
+
+ void appendExtractor(String rule, int sourceLine, ExecutionContext ctx);
+
+ /**
+ * Statement-level probe — opt-in via session granularity.
+ * {@code sourceText} is the verbatim DSL fragment for the line so the UI
+ * can render "line {sourceLine}: {sourceText}" without round-tripping to
+ * the source. Block-mode recorders early-return; the probe call site is
+ * unconditional in the bytecode so the choice stays recorder-local.
+ */
+ void appendLine(String rule, int sourceLine, String sourceText, ExecutionContext ctx);
+
+ /**
+ * Terminal record-output probe. {@code outputClass} is the typed builder
+ * class name (e.g. {@code LogBuilder}, {@code EnvoyAccessLogBuilder}); the
+ * recorder serialises the output's bean properties for UI rendering.
+ * Fires once per surviving record, AFTER the sink kept it.
+ */
+ void appendOutputRecord(String rule, int sourceLine, ExecutionContext ctx, String outputClass);
+
+ /**
+ * Terminal metric-output probe. {@code family} is the {@link SampleFamily}
+ * the LAL metric extractor produced for hand-off to MAL. The debugger
+ * does not chain into MAL from here — that's a separate session boundary.
+ */
+ void appendOutputMetric(String rule, int sourceLine, ExecutionContext ctx, SampleFamily family);
+}
diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsldebug/LalPayloadDebugDump.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsldebug/LalPayloadDebugDump.java
new file mode 100644
index 0000000000..c10db4954e
--- /dev/null
+++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsldebug/LalPayloadDebugDump.java
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.log.analyzer.v2.dsldebug;
+
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import com.google.protobuf.Message;
+import com.google.protobuf.MessageOrBuilder;
+import com.google.protobuf.util.JsonFormat;
+import org.apache.skywalking.apm.network.logging.v3.LogData;
+import org.apache.skywalking.oap.server.core.dsldebug.LogDataDebugDump;
+import org.apache.skywalking.oap.server.core.dsldebug.ToJson;
+
+/**
+ * Per-type renderers for LAL input/output payloads. All overloads are
+ * statically typed — callers hold the typed reference (or narrow it
+ * via {@code instanceof}) and the compiler picks the right overload.
+ * No runtime {@code Object} dispatcher: dispatch belongs at the call
+ * site where the static type is available.
+ *
+ * <p>Supported families:
+ * <ul>
+ * <li>{@link ToJson} — explicit opt-in. Custom POJO inputs / outputs
+ * implement this and call {@link #toJson(ToJson)}.</li>
+ * <li>{@link LogData} / {@link LogData.Builder} — native renderer
+ * that surfaces service / instance / endpoint / timestamp / tags
+ * / trace context plus a body-type-aware view of the body
+ * (TEXT / JSON / YAML).</li>
+ * <li>Any {@link Message} / {@link Message.Builder} — protobuf
+ * {@link JsonFormat} printer. Walks the proto descriptor (a
+ * structural, compile-time form, not Java reflection) and
+ * emits the canonical proto3 JSON shape.</li>
+ * </ul>
+ *
+ * <p>Each overload returns a JSON literal string suitable for inline
+ * embedding in a parent object payload.
+ *
+ * <p>Use {@link #unknownTypeHint(Object)} when none of the supported
+ * families match — the helper produces an operator-friendly hint
+ * object pointing at the missing {@link ToJson} implementation.
+ */
+public final class LalPayloadDebugDump {
+
+ private static final JsonFormat.Printer JSON_PRINTER =
+ JsonFormat.printer().omittingInsignificantWhitespace();
+
+ private LalPayloadDebugDump() {
+ }
+
+ /** Explicit opt-in renderer — delegates to the type's own contract. */
+ public static String toJson(final ToJson obj) {
+ if (obj == null) {
+ return "null";
+ }
+ final String value = obj.toJson();
+ return value == null ? "null" : value;
+ }
+
+ /** Native LogData renderer — body-type-aware (TEXT / JSON / YAML). */
+ public static String toJson(final LogData data) {
+ return LogDataDebugDump.toJson(data);
+ }
+
+ /** Convenience: builds the snapshot once and renders via {@link #toJson(LogData)}. */
+ public static String toJson(final LogData.Builder builder) {
+ return builder == null ? "null" : LogDataDebugDump.toJson(builder.build());
+ }
+
+ /** Generic protobuf renderer via {@link JsonFormat}. */
+ public static String toJson(final Message msg) {
+ if (msg == null) {
+ return "null";
+ }
+ return printProtoOrError(msg, msg.getDescriptorForType().getFullName());
+ }
+
+ /** Convenience: builds once and renders via {@link #toJson(Message)}. */
+ public static String toJson(final Message.Builder builder) {
+ return builder == null ? "null" : toJson(builder.build());
+ }
+
+ /**
+ * Defensive renderer for {@link MessageOrBuilder} — covers exotic
+ * proto wrappers that aren't a direct {@link Message} but expose
+ * the same printable shape.
+ */
+ public static String toJson(final MessageOrBuilder msg) {
+ if (msg == null) {
+ return "null";
+ }
+ return printProtoOrError(msg, msg.getClass().getSimpleName());
+ }
+
+ /**
+ * Operator-friendly hint object emitted by callers when their
+ * dispatch logic doesn't recognise the runtime type. The hint
+ * surfaces the FQCN so the operator knows where to add the
+ * {@link ToJson} implementation.
+ */
+ public static String unknownTypeHint(final Object obj) {
+ if (obj == null) {
+ return "null";
+ }
+ final JsonObject hint = new JsonObject();
+ hint.addProperty("type", obj.getClass().getName());
+ hint.addProperty("note", "requires-ToJson-impl");
+ return hint.toString();
+ }
+
+ /**
+ * Parse the result of any overload above into a {@link JsonElement}
+ * so a caller building a parent {@link JsonObject} can splice the
+ * result without going through a round-trip string parse.
+ */
+ public static JsonElement parse(final String json) {
+ return JsonParser.parseString(json);
+ }
+
+ private static String printProtoOrError(final MessageOrBuilder msg, final String typeLabel) {
+ try {
+ return JSON_PRINTER.print(msg);
+ } catch (final Exception e) {
+ final JsonObject obj = new JsonObject();
+ obj.addProperty("type", typeLabel);
+ obj.addProperty("error", "jsonformat-failed");
+ obj.addProperty("detail", e.getMessage());
+ return obj.toString();
+ }
+ }
+}
diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsldebug/LalStaticBindingHook.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsldebug/LalStaticBindingHook.java
new file mode 100644
index 0000000000..f6181139f9
--- /dev/null
+++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsldebug/LalStaticBindingHook.java
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.log.analyzer.v2.dsldebug;
+
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.skywalking.oap.log.analyzer.v2.dsl.LalExpression;
+import org.apache.skywalking.oap.server.core.dsldebug.GateHolder;
+
+/**
+ * Process-wide hook that publishes static-loader LAL rule bindings into the
+ * dsl-debugging session registry. Same shape as
+ * {@code MalStaticBindingHook}: dsl-debugging installs a real sink at boot
+ * (no-op default), each LAL static-loader site calls
+ * {@link #publish(String, String, LalExpression)} once per compiled rule.
+ *
+ * <h2>Workflow</h2>
+ * <pre>
+ * dsl-debugging prepare() ─► LalStaticBindingHook.install(sink)
+ *
+ * LogFilterListener.Factory.loadStaticRules:
+ * for each LALConfig c:
+ * CompiledLAL compiled = compile(c);
+ * LalStaticBindingHook.publish(c.getSourceName(), c.getName(), compiled.dsl.getExpression());
+ * │ │ │
+ * fileName ruleName holder
+ * (matches the runtime-rule LAL key shape)
+ * </pre>
+ *
+ * <p>The registry's RuleKey for LAL is {@code (LAL, fileName, ruleName)};
+ * passing the file's {@code sourceName} keeps static and runtime entries
+ * keyed identically so a runtime-rule replace simply puts over the static
+ * binding without orphaning it.
+ */
+public final class LalStaticBindingHook {
+
+ /** Sink the hook forwards every bound holder to. */
+ @FunctionalInterface
+ public interface Sink {
+ void publish(String fileName, String ruleName, GateHolder holder);
+ }
+
+ private static final Sink NOOP = (fileName, ruleName, holder) -> {
+ };
+
+ private static final AtomicReference<Sink> SINK = new AtomicReference<>(NOOP);
+
+ private LalStaticBindingHook() {
+ }
+
+ public static void install(final Sink s) {
+ SINK.set(s == null ? NOOP : s);
+ }
+
+ public static void reset() {
+ SINK.set(NOOP);
+ }
+
+ /**
+ * Publish one LAL rule's debug holder. No-op when any argument is null
+ * or the expression carries no holder (defensive — LalExpression guarantees
+ * a non-null holder, but the check keeps the static-loader path
+ * crash-free against any future refactor).
+ */
+ public static void publish(final String fileName, final String ruleName,
+ final LalExpression expression) {
+ if (fileName == null || ruleName == null || expression == null) {
+ return;
+ }
+ final GateHolder holder = expression.debugHolder();
+ if (holder == null) {
+ return;
+ }
+ SINK.get().publish(fileName, ruleName, holder);
+ }
+}
diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/provider/LogAnalyzerModuleProvider.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/provider/LogAnalyzerModuleProvider.java
index 37ef9c8d5e..cc48a83cfd 100644
--- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/provider/LogAnalyzerModuleProvider.java
+++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/provider/LogAnalyzerModuleProvider.java
@@ -28,6 +28,7 @@ import org.apache.skywalking.oap.log.analyzer.v2.provider.log.LogAnalyzerService
import org.apache.skywalking.oap.log.analyzer.v2.provider.log.listener.LogFilterListener;
import org.apache.skywalking.oap.meter.analyzer.v2.MalConverterRegistry;
import org.apache.skywalking.oap.meter.analyzer.v2.MetricConvert;
+import org.apache.skywalking.oap.meter.analyzer.v2.dsldebug.MalStaticBindingHook;
import org.apache.skywalking.oap.server.configuration.api.ConfigurationModule;
import org.apache.skywalking.oap.server.core.CoreModule;
import org.apache.skywalking.oap.server.core.analysis.meter.MeterSystem;
@@ -156,7 +157,10 @@ public class LogAnalyzerModuleProvider extends ModuleProvider {
// Use the catalog:name key convention so a runtime-rule /addOrUpdate for the same
// (catalog, name) replaces this static entry in place instead of running two
// converters against the same sample stream.
- addOrReplaceMetricConvert("log-mal-rules:" + rule.getName(), new MetricConvert(rule, meterSystem));
+ final MetricConvert convert = new MetricConvert(rule, meterSystem);
+ addOrReplaceMetricConvert("log-mal-rules:" + rule.getName(), convert);
+ // Static-loader debug-binding publish — no-op when dsl-debugging is disabled.
+ MalStaticBindingHook.publish("log-mal-rules", rule.getName(), convert);
}
try {
// Light up the Factory now that all peer modules are past prepare:
diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/provider/log/listener/LogFilterListener.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/provider/log/listener/LogFilterListener.java
index c3f4af9265..6568b10bab 100644
--- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/provider/log/listener/LogFilterListener.java
+++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/provider/log/listener/LogFilterListener.java
@@ -30,12 +30,16 @@ import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
import java.util.HashMap;
-import org.apache.skywalking.oap.log.analyzer.v2.dsl.ExecutionContext;
import org.apache.skywalking.oap.log.analyzer.v2.dsl.DSL;
+import org.apache.skywalking.oap.log.analyzer.v2.dsl.ExecutionContext;
+import org.apache.skywalking.oap.log.analyzer.v2.dsldebug.LalStaticBindingHook;
import org.apache.skywalking.oap.log.analyzer.v2.provider.LALConfig;
import org.apache.skywalking.oap.log.analyzer.v2.provider.LALConfigs;
import org.apache.skywalking.oap.log.analyzer.v2.provider.LogAnalyzerModuleConfig;
+import com.google.protobuf.Message;
+import org.apache.skywalking.apm.network.logging.v3.LogData;
import org.apache.skywalking.oap.log.analyzer.v2.spi.LALSourceTypeProvider;
+import org.apache.skywalking.oap.server.core.dsldebug.ToJson;
import org.apache.skywalking.oap.server.core.analysis.Layer;
import org.apache.skywalking.oap.server.core.source.LALOutputBuilder;
@@ -62,11 +66,23 @@ import org.apache.skywalking.oap.server.library.module.Service;
public class LogFilterListener implements LogAnalysisListener {
private final List<DSL> dsls;
private final boolean autoMode;
+ /**
+ * Resolved SPI provider for this listener's layer (null for the
+ * default agent path). Threaded onto every {@link ExecutionContext}
+ * the listener creates so the compiler / runtime can read the
+ * layer's declared input and output types. Payload rendering for
+ * dsl-debugging is owned by the rule's
+ * {@link org.apache.skywalking.oap.server.core.source.LALOutputBuilder},
+ * not the provider.
+ */
+ private final LALSourceTypeProvider sourceTypeProvider;
private List<ExecutionContext> contexts;
- LogFilterListener(final Collection<DSL> dsls, final boolean autoMode) {
+ LogFilterListener(final Collection<DSL> dsls, final boolean autoMode,
+ final LALSourceTypeProvider sourceTypeProvider) {
this.dsls = new ArrayList<>(dsls);
this.autoMode = autoMode;
+ this.sourceTypeProvider = sourceTypeProvider;
}
@Override
@@ -100,6 +116,7 @@ public class LogFilterListener implements LogAnalysisListener {
contexts = new ArrayList<>(dsls.size());
for (int i = 0; i < dsls.size(); i++) {
final ExecutionContext ctx = new ExecutionContext().init(metadata, input);
+ ctx.setSourceTypeProvider(sourceTypeProvider);
if (autoMode) {
ctx.autoLayerMode(true);
}
@@ -178,6 +195,7 @@ public class LogFilterListener implements LogAnalysisListener {
// the JDK's {@code ServiceLoader} — no moduleManager.find required, safe in prepare.
this.spiProviders = new HashMap<>();
for (final LALSourceTypeProvider p : ServiceLoader.load(LALSourceTypeProvider.class)) {
+ validateInputTypeContract(p);
spiProviders.put(p.layer(), p);
log.info("LALSourceTypeProvider: layer={}, inputType={}, outputType={}",
p.layer().name(), p.inputType().getName(),
@@ -221,6 +239,10 @@ public class LogFilterListener implements LogAnalysisListener {
"Layer " + compiled.layer.name() + " has already set " + c.getName() + " rule.");
}
}
+ // Publish per-rule debug holder into the dsl-debugging registry. Static and
+ // runtime-rule entries share the same (LAL, fileName, ruleName) key shape so
+ // a runtime-rule replace puts over the static binding without orphaning it.
+ LalStaticBindingHook.publish(c.getSourceName(), c.getName(), compiled.dsl.getExpression());
}
// Publish: readers from now on see the startup-complete registry.
this.dsls = initDsls;
@@ -282,6 +304,27 @@ public class LogFilterListener implements LogAnalysisListener {
}
}
+ /**
+ * Look up the live {@link DSL} for {@code (layer, ruleName)}. Returns {@code null}
+ * when the key is not currently registered. {@code layer == null} routes to the
+ * auto-layer table the same way {@link #remove(Layer, String)} does.
+ *
+ * <p>Read-only — exposed so the dsl-debugging module can resolve a rule's
+ * {@code GateHolder} via the live DSL right after the LAL apply path commits.
+ * No write lock; the volatile {@code dsls} / {@code autoDsls} fields are
+ * publication-safe for snapshot reads.
+ */
+ public DSL find(final Layer layer, final String ruleName) {
+ if (ruleName == null) {
+ return null;
+ }
+ if (layer == null) {
+ return autoDsls.get(ruleName);
+ }
+ final Map<String, DSL> layerMap = dsls.get(layer);
+ return layerMap == null ? null : layerMap.get(ruleName);
+ }
+
/** Runtime remove. No-op when the key isn't present. */
public void remove(final Layer layer, final String ruleName) {
synchronized (writeLock) {
@@ -382,6 +425,38 @@ public class LogFilterListener implements LogAnalysisListener {
}
}
+ /**
+ * Boot-time validation: every {@link LALSourceTypeProvider#inputType()}
+ * must satisfy one of the LAL-supported input families — {@code LogData} /
+ * {@code LogData.Builder} (default agent path), a protobuf {@code Message}
+ * subclass (typed receiver inputs), or a class implementing {@link ToJson}
+ * (custom POJO inputs). A provider that declares an out-of-family
+ * {@code inputType()} will fail downstream dispatch (the framework
+ * fallback {@code (LogData.Builder)} cast throws ClassCastException at
+ * runtime), so we surface the contract violation here at startup with a
+ * pointer at the SPI rather than letting it leak into the receiver hot path.
+ */
+ private static void validateInputTypeContract(final LALSourceTypeProvider provider) {
+ final Class<?> inputType = provider.inputType();
+ if (inputType == null) {
+ return;
+ }
+ if (LogData.class.isAssignableFrom(inputType)
+ || LogData.Builder.class.isAssignableFrom(inputType)
+ || Message.class.isAssignableFrom(inputType)
+ || Message.Builder.class.isAssignableFrom(inputType)
+ || ToJson.class.isAssignableFrom(inputType)) {
+ return;
+ }
+ throw new IllegalStateException(
+ "LALSourceTypeProvider " + provider.getClass().getName() + " for layer "
+ + provider.layer() + " declared inputType " + inputType.getName()
+ + " — LAL accepts only LogData / LogData.Builder (default agent path), "
+ + "a protobuf Message subclass, or a type implementing "
+ + ToJson.class.getName() + ". Update the provider's inputType() or "
+ + "implement ToJson on " + inputType.getName() + ".");
+ }
+
private static Class<?> resolveInputType(final LALConfig config,
final LALSourceTypeProvider spiProvider) throws ModuleStartException {
final String yamlType = config.getInputType();
@@ -476,7 +551,8 @@ public class LogFilterListener implements LogAnalysisListener {
if (eligible.isEmpty()) {
return null;
}
- return new LogFilterListener(eligible, true);
+ // auto-mode crosses layers — no single SPI provider applies
+ return new LogFilterListener(eligible, true, null);
}
final Map<String, DSL> dsl = dsls.get(layer);
if (dsl == null) {
@@ -488,7 +564,7 @@ public class LogFilterListener implements LogAnalysisListener {
if (eligible.isEmpty()) {
return null;
}
- return new LogFilterListener(eligible, false);
+ return new LogFilterListener(eligible, false, spiProviders.get(layer));
}
private static Collection<DSL> filterSuspended(final Map<String, DSL> source,
diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/provider/log/listener/RecordSinkListener.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/provider/log/listener/RecordSinkListener.java
index e65a9031ce..17334e1518 100644
--- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/provider/log/listener/RecordSinkListener.java
+++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/provider/log/listener/RecordSinkListener.java
@@ -20,8 +20,10 @@ package org.apache.skywalking.oap.log.analyzer.v2.provider.log.listener;
import lombok.SneakyThrows;
import org.apache.skywalking.oap.log.analyzer.v2.dsl.ExecutionContext;
+import org.apache.skywalking.oap.log.analyzer.v2.dsldebug.LALDebug;
import org.apache.skywalking.oap.log.analyzer.v2.provider.LogAnalyzerModuleConfig;
import org.apache.skywalking.oap.server.core.CoreModule;
+import org.apache.skywalking.oap.server.core.dsldebug.GateHolder;
import org.apache.skywalking.oap.server.core.source.LALOutputBuilder;
import org.apache.skywalking.oap.server.core.source.LogMetadata;
import org.apache.skywalking.oap.server.core.source.SourceReceiver;
@@ -91,6 +93,15 @@ public class RecordSinkListener implements LogSinkListener {
}
builder = ctx.outputAsBuilder();
builder.init(metadata, input, moduleManager);
+ // Terminal-output probe: typed builder is fully populated, complete()
+ // hasn't fired yet. The probe pulls the rule's debug holder off ctx so
+ // this wrapper layer doesn't need its own LalExpression reference.
+ // Fires only for records the sink kept (parse() is bypassed otherwise).
+ final GateHolder debugHolder = ctx.getDebugHolder();
+ if (debugHolder != null && debugHolder.isGateOn()) {
+ LALDebug.captureOutputRecord(
+ debugHolder, "", 0, ctx, builder.getClass().getSimpleName());
+ }
return this;
}
diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/spi/LALSourceTypeProvider.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/spi/LALSourceTypeProvider.java
index f97f56c044..a33f05887f 100644
--- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/spi/LALSourceTypeProvider.java
+++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/spi/LALSourceTypeProvider.java
@@ -65,6 +65,22 @@ public interface LALSourceTypeProvider {
* The Java type passed as {@code input} by the receiver plugin for
* this layer. The compiler resolves getter chains on this type at
* compile time.
+ *
+ * <p>The returned class must satisfy one of the LAL-supported
+ * input families:
+ * <ul>
+ * <li>{@code LogData} or {@code LogData.Builder} — the standard
+ * agent-side input shape (handled by the framework with no
+ * provider override needed).</li>
+ * <li>A {@code com.google.protobuf.Message} subclass — typed
+ * receiver inputs like {@code HTTPAccessLogEntry}.</li>
+ * <li>A class implementing
+ * {@link org.apache.skywalking.oap.server.core.dsldebug.ToJson}
+ * — explicit opt-in for custom POJO inputs.</li>
+ * </ul>
+ * The framework validates this at boot — providers declaring
+ * an out-of-family {@code inputType()} fail OAP startup with a
+ * pointer at this contract.
*/
Class<?> inputType();
diff --git a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALClassGeneratorTestBase.java b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALClassGeneratorTestBase.java
index 2c22a9f9da..29ba35d43c 100644
--- a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALClassGeneratorTestBase.java
+++ b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALClassGeneratorTestBase.java
@@ -42,6 +42,15 @@ abstract class LALClassGeneratorTestBase {
@BeforeEach
void setUp(final TestInfo testInfo) {
+ // Mirror the production toggle so a developer can rebuild with
+ // SW_DSL_DEBUGGING_INJECTION_ENABLED=true and inspect the
+ // debug-probe-injected generated classes (gate field +
+ // probe call sites) in target/lal-generated-classes/.
+ if ("true".equalsIgnoreCase(System.getenv("SW_DSL_DEBUGGING_INJECTION_ENABLED"))) {
+ org.apache.skywalking.oap.server.core.dsldebug.DSLDebugCodegenSwitch.enableInjection();
+ } else {
+ org.apache.skywalking.oap.server.core.dsldebug.DSLDebugCodegenSwitch.resetInjection();
+ }
generator = new LALClassGenerator(new ClassPool(true));
generator.setClassOutputDir(DslClassOutput.unitTestDir("lal"));
// Build hint from test class + method for readable .class file names
diff --git a/oap-server/analyzer/meter-analyzer/CLAUDE.md b/oap-server/analyzer/meter-analyzer/CLAUDE.md
index 047cd1e627..fc79eabb8e 100644
--- a/oap-server/analyzer/meter-analyzer/CLAUDE.md
+++ b/oap-server/analyzer/meter-analyzer/CLAUDE.md
@@ -241,14 +241,19 @@ Metadata is extracted statically from the AST at compile time by `MALMetadataExt
## Debug Output
-When `SW_DYNAMIC_CLASS_ENGINE_DEBUG=true` environment variable is set, generated `.class` files are written to disk for inspection:
+When `SW_DYNAMIC_CLASS_ENGINE_DEBUG=true` environment variable is set, generated `.class` files are written to disk for inspection. Each `.class` is paired with a `<ClassName>.java` sidecar — the verbatim Java source the codegen fed Javassist:
```
{skywalking}/mal-rt/
*.class — Main MalExpression class per expression
+ *.java — Javassist compile input (synthetic; for IDE source-attach)
*$_tag.class — Companion class per closure (one per tag/forEach/instance/decorate call)
```
+The `.java` sidecar exists so IDE source-attach renders the actual codegen input directly without relying on FernFlower / a decompiler. Javassist-emitted bytecode often confuses decompilers (no `goto` consolidation, slot reuse with mixed types, debug-injected `if (gate.isGateOn()) { ... }` chains), and FernFlower frequently bails to "compiled code" stubs. The source-attach path always works and shows the EXACT code Javassist compiled — gate fields, probe call sites, closure dispatch, the lot.
+
+When `SW_DSL_DEBUGGING_INJECTION_ENABLED=true` is also set, the codegen emits the per-rule `GateHolder debug` field plus `MALDebug.captureXxx(...)` probe sites at every chain stage. Both `.class` and `.java` reflect the with-debug shape. The two env vars are independent: `SW_DYNAMIC_CLASS_ENGINE_DEBUG` controls disk dump; `SW_DSL_DEBUGGING_INJECTION_ENABLED` controls codegen branch.
+
This is the same env variable used by OAL. Useful for debugging code generation issues or comparing V1 vs V2 output. In tests, use `setClassOutputDir(dir)` instead.
## MAL Input Data Mock Principles
diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/Analyzer.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/Analyzer.java
index 3186609c32..c303c72a7d 100644
--- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/Analyzer.java
+++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/Analyzer.java
@@ -43,8 +43,10 @@ import org.apache.skywalking.oap.meter.analyzer.v2.dsl.FilterExpression;
import org.apache.skywalking.oap.meter.analyzer.v2.dsl.Result;
import org.apache.skywalking.oap.meter.analyzer.v2.dsl.Sample;
import org.apache.skywalking.oap.meter.analyzer.v2.dsl.SampleFamily;
+import org.apache.skywalking.oap.meter.analyzer.v2.dsldebug.MALDebug;
import org.apache.skywalking.oap.server.core.analysis.Layer;
import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.dsldebug.GateHolder;
import org.apache.skywalking.oap.server.core.analysis.manual.endpoint.EndpointTraffic;
import org.apache.skywalking.oap.server.core.analysis.manual.instance.InstanceTraffic;
import org.apache.skywalking.oap.server.core.analysis.manual.relation.process.ProcessRelationClientSideMetrics;
@@ -222,6 +224,7 @@ public class Analyzer {
private final FilterExpression filterExpression;
+ @Getter
private final Expression expression;
private final MeterSystem meterSystem;
@@ -268,8 +271,37 @@ public class Analyzer {
}
return;
}
+ // Resolve the rule's capture binding once per ingest pass. When dsl-debugging
+ // codegen injection is off the expression returns null (no probe field on the
+ // generated class); the hand-written probe sites in this method must therefore
+ // null-check before consulting the gate, otherwise an injection-off cluster
+ // NPEs on the very first MAL pass and silently drops every metric.
+ final GateHolder debugHolder = expression.debugHolder();
+ final boolean debugOn = debugHolder != null && debugHolder.isGateOn();
+ // Per-execution boundary: one analysis pass = one captured record.
+ // begin marks the start (flushing any prior orphan); end fires in
+ // finally so a thrown exception still publishes the partial record.
+ if (debugOn) {
+ MALDebug.captureBeginAnalysis(debugHolder, metricName);
+ }
+ try {
if (filterExpression != null) {
+ // File-level filter probe: capture the FULL post-filter map keyed
+ // by metric name. A rule that combines `metric_a + metric_b + ...`
+ // shows up to N entries here — every metric whose samples
+ // survived the filter, plus none of the ones that filtered to
+ // EMPTY. Pre-filter view is passed as the captured map only when
+ // the filter short-circuits the rule (kept=false), so the
+ // operator can see which families went in but didn't make it out.
+ final Map<String, SampleFamily> preFilter = input;
input = filterExpression.filter(input);
+ if (debugOn) {
+ final boolean kept = !input.isEmpty();
+ MALDebug.captureFilter(debugHolder, metricName,
+ filterExpression.getLiteral(),
+ kept ? input : preFilter,
+ kept);
+ }
if (input.isEmpty()) {
if (log.isDebugEnabled()) {
log.debug("{} is ignored due to mismatch of filter {}", expression, filterExpression);
@@ -294,12 +326,18 @@ public class Analyzer {
}
SampleFamily.RunningContext ctx = r.getData().context;
Map<MeterEntity, Sample[]> meterSamples = ctx.getMeterSamples();
+ // No `scope` probe here — the verbatim scope-binding DSL is already captured by the
+ // preceding chain `stage` record (e.g. "service(['service_name'], Layer.GENERAL)"),
+ // and the per-entity grouping is observable from the meterBuild records' payloads.
+ // A separate scope record would carry only the metric type ("single"/"labeled"/…),
+ // which is internal classification, not anything the operator authored.
meterSamples.forEach((meterEntity, ss) -> {
generateTraffic(meterEntity);
switch (metricType) {
case single:
AcceptableValue<Long> sv = meterSystem.buildMetrics(metricName, Long.class);
sv.accept(meterEntity, getValue(ss[0]));
+ captureMeterEmit(debugHolder, meterEntity, sv, ss[0].getTimestamp());
send(sv, ss[0].getTimestamp());
break;
case labeled:
@@ -312,6 +350,7 @@ public class Analyzer {
dt.put(dataLabel, getValue(each));
}
lv.accept(meterEntity, dt);
+ captureMeterEmit(debugHolder, meterEntity, lv, ss[0].getTimestamp());
send(lv, ss[0].getTimestamp());
break;
case histogram:
@@ -341,17 +380,49 @@ public class Analyzer {
AcceptableValue<BucketedValues> v = meterSystem.buildMetrics(
metricName, BucketedValues.class);
v.accept(meterEntity, bv);
+ captureMeterEmit(debugHolder, meterEntity, v, time);
send(v, time);
return;
}
AcceptableValue<PercentileArgument> v = meterSystem.buildMetrics(
metricName, PercentileArgument.class);
v.accept(meterEntity, new PercentileArgument(bv, percentiles));
+ captureMeterEmit(debugHolder, meterEntity, v, time);
send(v, time);
});
break;
}
});
+ } finally {
+ if (debugOn) {
+ MALDebug.captureEndAnalysis(debugHolder, metricName);
+ }
+ }
+ }
+
+ /**
+ * Fires the {@code captureMeterBuild} + {@code captureMeterEmit} probe pair right
+ * before {@code send(...)} hands the value to {@code MeterSystem.doStreamingCalculation}
+ * — i.e., the L1-bound boundary that ends MAL's debugger scope. Pulled into one helper
+ * so the four metric-type branches above share one capture shape (no copy-pasted
+ * gate check, no drift between branches).
+ */
+ /**
+ * Terminal capture for a meter: by the time we reach this site, the typed
+ * {@code AcceptableValue} has been constructed and {@code accept()}-populated
+ * in one step, so a separate "build" probe would duplicate the same value
+ * state. OAL splits build vs aggregation because its entry function mutates
+ * the {@code Metrics} between them; MAL has nothing equivalent.
+ */
+ private void captureMeterEmit(final GateHolder debugHolder,
+ final MeterEntity entity,
+ final AcceptableValue<?> value,
+ final long sourceTimestamp) {
+ if (debugHolder == null || !debugHolder.isGateOn()) {
+ return;
+ }
+ MALDebug.captureMeterEmit(debugHolder, metricName, entity, metricName,
+ value, TimeBucket.getMinuteTimeBucket(sourceTimestamp));
}
private long getValue(Sample sample) {
diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/MetricConvert.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/MetricConvert.java
index cd86c14240..b3539b2975 100644
--- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/MetricConvert.java
+++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/MetricConvert.java
@@ -23,8 +23,10 @@ import com.google.common.base.Strings;
import com.google.common.collect.ImmutableMap;
import io.vavr.control.Try;
import java.util.Collections;
+import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.List;
+import java.util.Map;
import java.util.Set;
import java.util.StringJoiner;
import java.util.stream.IntStream;
@@ -35,6 +37,7 @@ import org.apache.commons.lang3.StringUtils;
import org.apache.skywalking.oap.meter.analyzer.v2.dsl.FilterExpression;
import org.apache.skywalking.oap.meter.analyzer.v2.dsl.SampleFamily;
import org.apache.skywalking.oap.server.core.analysis.meter.MeterSystem;
+import org.apache.skywalking.oap.server.core.dsldebug.GateHolder;
import org.apache.skywalking.oap.server.core.storage.model.StorageManipulationOpt;
import static java.util.stream.Collectors.toList;
@@ -131,7 +134,7 @@ public class MetricConvert {
final MetricRuleConfig.RuleConfig r = rules.get(i);
final String yamlSource = sourceName != null
? sourceName + ".yaml:" + i : null;
- return prepareAnalyzer(
+ final Analyzer analyzer = prepareAnalyzer(
formatMetricName(rule, r.getName()),
filter,
formatExp(rule.getExpPrefix(), rule.getExpSuffix(), r.getExp()),
@@ -141,6 +144,23 @@ public class MetricConvert {
targetClassLoader,
storageOpt
);
+ // Stamp the rule's structured fields onto the per-metric
+ // GateHolder so dsl-debugging records render the rule as
+ // {metricPrefix, name, filter, exp, expSuffix} alongside
+ // the verbatim DSL. Only effective when codegen injection
+ // is enabled (otherwise debugHolder() returns null).
+ final GateHolder holder = analyzer.getExpression().debugHolder();
+ if (holder != null) {
+ final LinkedHashMap<String, String> meta = new LinkedHashMap<>();
+ putIfNonEmpty(meta, "metricPrefix", rule.getMetricPrefix());
+ putIfNonEmpty(meta, "name", r.getName());
+ putIfNonEmpty(meta, "filter", rule.getFilter());
+ putIfNonEmpty(meta, "exp", r.getExp());
+ putIfNonEmpty(meta, "expSuffix", rule.getExpSuffix());
+ putIfNonEmpty(meta, "expPrefix", rule.getExpPrefix());
+ holder.setMetadata(meta);
+ }
+ return analyzer;
}
).collect(toList());
// Phase 2 — register. Track each metric name as it's successfully registered so a
@@ -172,6 +192,16 @@ public class MetricConvert {
@Getter
private final Set<String> registeredMetricNames;
+ /**
+ * Snapshot of every analyzer this convert owns, exposed for debug-API integration so the
+ * dsl-debugging module can iterate {@code (metricName, debugHolder)} pairs without
+ * reaching into the {@link Analyzer} internals. Read-only — mutating the list would
+ * corrupt the converter's broadcast loop.
+ */
+ public List<Analyzer> getAnalyzers() {
+ return Collections.unmodifiableList(analyzers);
+ }
+
/**
* Thrown from the ctor when phase-2 register throws after at least one metric was already
* registered. Carries the subset that did land, so the caller can unregister exactly what
@@ -224,6 +254,13 @@ public class MetricConvert {
* every rule's MAL expression is parsed + typed before any {@code MeterSystem.create}
* call fires. Phase 2 runs {@link Analyzer#register} on the returned objects.
*/
+ private static void putIfNonEmpty(final Map<String, String> target,
+ final String key, final String value) {
+ if (value != null && !value.isEmpty()) {
+ target.put(key, value);
+ }
+ }
+
Analyzer prepareAnalyzer(final String metricsName,
final FilterExpression filter,
final String exp,
diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALBytecodeHelper.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALBytecodeHelper.java
index e8291917ca..7f7322d5c6 100644
--- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALBytecodeHelper.java
+++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALBytecodeHelper.java
@@ -196,6 +196,39 @@ final class MALBytecodeHelper {
}
}
+ /**
+ * Writes the Javassist-input Java source for a generated class as a
+ * sibling {@code <ClassName>.java} file when {@code classOutputDir} is
+ * set. The {@code SourceFile} attribute on the {@code .class} points at
+ * this name, so IDE source-attach renders it directly without going
+ * through FernFlower / a decompiler — operators see the EXACT code
+ * Javassist compiled. Caller passes pre-formatted Java source built
+ * during codegen (the {@code makeClass(...)}/{@code make(...)} input
+ * strings concatenated with the public class envelope).
+ */
+ void writeSourceFile(final CtClass ctClass, final String javaSource) {
+ if (classOutputDir == null || javaSource == null) {
+ return;
+ }
+ if (!classOutputDir.exists()) {
+ classOutputDir.mkdirs();
+ }
+ final File file = new File(
+ classOutputDir, ctClass.getSimpleName() + ".java");
+ try (java.io.FileWriter w = new java.io.FileWriter(file)) {
+ w.write("// Synthetic source — Javassist compile input for ");
+ w.write(ctClass.getSimpleName());
+ w.write("\n// Written when SW_DYNAMIC_CLASS_ENGINE_DEBUG is on; used by IDE\n");
+ w.write("// source-attach to render the bytecode without FernFlower.\n\n");
+ w.write(javaSource);
+ if (!javaSource.endsWith("\n")) {
+ w.write("\n");
+ }
+ } catch (Exception e) {
+ log.warn("Failed to write source file {}: {}", file, e.getMessage(), e);
+ }
+ }
+
// ==================== Bytecode attributes ====================
/**
diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALClassGenerator.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALClassGenerator.java
index 1310ca0f55..5fdb8c273a 100644
--- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALClassGenerator.java
+++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALClassGenerator.java
@@ -106,6 +106,15 @@ public final class MALClassGenerator {
this.bytecodeHelper.setPerFileClassLoader(targetClassLoader != null);
}
+ /**
+ * Optional content hash threaded into every generated rule's {@code GateHolder}
+ * constructor argument. Stamped onto every captured debug record so a
+ * UI / CLI session can detect mid-session hot-update boundaries. Defaults
+ * to the empty string when callers don't supply one (boot-time loaders that
+ * don't track per-rule hashes).
+ */
+ private String content = "";
+
public void setClassOutputDir(final File dir) {
bytecodeHelper.setClassOutputDir(dir);
}
@@ -118,6 +127,18 @@ public final class MALClassGenerator {
bytecodeHelper.setYamlSource(yamlSource);
}
+ /**
+ * Sets the content hash baked into the next compiled rule's
+ * {@code GateHolder} constructor argument. Caller-supplied so the
+ * runtime-rule hot-update path can pass its already-computed
+ * {@code ContentHash} value through unchanged. Pass {@code null} or
+ * the empty string to skip the stamp — the generated holder still works,
+ * but captured records carry an empty hash.
+ */
+ public void setContent(final String content) {
+ this.content = content == null ? "" : content;
+ }
+
// Package-private accessors for MALClosureCodegen back-reference
MALBytecodeHelper getBytecodeHelper() {
return bytecodeHelper;
@@ -186,12 +207,24 @@ public final class MALClassGenerator {
? bytecodeHelper.getClassNameHint() : "filter"));
bytecodeHelper.writeClassFile(ctClass);
+ bytecodeHelper.writeSourceFile(ctClass, wrapMalFilterSource(ctClass, filterBody));
final Class<?> clazz = defineClass(ctClass);
ctClass.detach();
return (MalFilter) clazz.getDeclaredConstructor().newInstance();
}
+ /** Wraps the filter test() method body as a compilable class envelope. */
+ private static String wrapMalFilterSource(final CtClass ctClass, final String filterBody) {
+ final StringBuilder sb = new StringBuilder();
+ sb.append("package ").append(ctClass.getPackageName()).append(";\n\n");
+ sb.append("public class ").append(ctClass.getSimpleName())
+ .append(" implements org.apache.skywalking.oap.meter.analyzer.v2.dsl.MalFilter {\n\n");
+ sb.append(" ").append(filterBody.replace("\n", "\n ")).append("\n");
+ sb.append("}\n");
+ return sb.toString();
+ }
+
/**
* Loads a generated class through the configured {@link #targetClassLoader} when set
* (runtime-rule hot-update path: class lands in the per-file {@code RuleClassLoader}),
@@ -281,8 +314,15 @@ public final class MALClassGenerator {
ctClass.addConstructor(CtNewConstructor.defaultConstructor(ctClass));
- // 3. Generate run() method
- final MALExprCodegen exprCodegen = new MALExprCodegen(closureFieldNames);
+ // 2.5. Emit the per-rule GateHolder instance field + accessor so the
+ // run() body's gate-checks have a target to read. Field is final +
+ // initialized inline so the value is stamped at construction; the
+ // accessor returns the same instance every call (no per-call alloc).
+ emitDebugHolderMembers(ctClass);
+
+ // 3. Generate run() method — pass metricName as ruleName so probe call sites
+ // emitted by MALExprCodegen and MALMethodChainCodegen carry it as their first arg.
+ final MALExprCodegen exprCodegen = new MALExprCodegen(closureFieldNames, metricName);
final String runBody = exprCodegen.generateRunMethod(ast);
// 4. Extract metadata
@@ -327,6 +367,8 @@ public final class MALClassGenerator {
}
bytecodeHelper.writeClassFile(ctClass);
+ bytecodeHelper.writeSourceFile(ctClass, wrapMalExpressionSource(
+ ctClass, closureFieldNames, closureInterfaceTypes, runBody, metadataBody));
final Class<?> clazz = defineClass(ctClass);
ctClass.detach();
@@ -334,6 +376,63 @@ public final class MALClassGenerator {
return (MalExpression) clazz.getDeclaredConstructor().newInstance();
}
+ /**
+ * Wraps the run() / metadata() method bodies plus closure field stubs
+ * + GateHolder field as a compilable class envelope.
+ */
+ private static String wrapMalExpressionSource(
+ final CtClass ctClass,
+ final List<String> closureFieldNames,
+ final List<String> closureInterfaceTypes,
+ final String runBody,
+ final String metadataBody) {
+ final StringBuilder sb = new StringBuilder();
+ sb.append("package ").append(ctClass.getPackageName()).append(";\n\n");
+ sb.append("public class ").append(ctClass.getSimpleName())
+ .append(" implements org.apache.skywalking.oap.meter.analyzer.v2.dsl.MalExpression {\n\n");
+ for (int i = 0; i < closureFieldNames.size(); i++) {
+ sb.append(" public static final ").append(closureInterfaceTypes.get(i))
+ .append(' ').append(closureFieldNames.get(i)).append(";\n");
+ }
+ if (org.apache.skywalking.oap.server.core.dsldebug.DSLDebugCodegenSwitch.isInjectionEnabled()) {
+ sb.append(" public final org.apache.skywalking.oap.server.core.dsldebug.GateHolder debug;\n");
+ sb.append(" public org.apache.skywalking.oap.server.core.dsldebug.GateHolder debugHolder() { return this.debug; }\n");
+ }
+ sb.append("\n ").append(runBody.replace("\n", "\n ")).append("\n\n");
+ sb.append(" ").append(metadataBody.replace("\n", "\n ")).append("\n");
+ sb.append("}\n");
+ return sb.toString();
+ }
+
+ /**
+ * Adds the {@code public final GateHolder debug} instance field and
+ * the {@code debugHolder()} accessor on the generated rule class.
+ * Both are mandatory for every compiled rule because the {@code run()}
+ * codegen embeds {@code if (this.debug.isGateOn()) MALDebug.captureXxx(this.debug, ...)}
+ * call sites that read the field directly. The constructor arg is the
+ * configured {@link #content}, escaped as a Java string literal.
+ */
+ private void emitDebugHolderMembers(final CtClass ctClass) throws javassist.CannotCompileException {
+ if (!org.apache.skywalking.oap.server.core.dsldebug.DSLDebugCodegenSwitch.isInjectionEnabled()) {
+ // Injection off — generated class inherits MalExpression.debugHolder()'s
+ // default null return, no GateHolder field is emitted, and the captureXxx
+ // call sites the helpers below would emit are also skipped. The compiled
+ // bytecode is byte-identical to a build without SWIP-13.
+ return;
+ }
+ final String holderFqcn = "org.apache.skywalking.oap.server.core.dsldebug.GateHolder";
+ final String escapedContent = MALCodegenHelper.escapeJava(content);
+ // Per-rule capture binding — instance field, lowercase per Java
+ // convention (it's a final but not a static-final constant).
+ ctClass.addField(javassist.CtField.make(
+ "public final " + holderFqcn + " debug = new " + holderFqcn
+ + "(\"" + escapedContent + "\");",
+ ctClass));
+ ctClass.addMethod(CtNewMethod.make(
+ "public " + holderFqcn + " debugHolder() { return this.debug; }",
+ ctClass));
+ }
+
// ==================== Debug/source generation ====================
/**
@@ -357,6 +456,8 @@ public final class MALClassGenerator {
fieldNames.add("_" + suffix);
}
+ // Source view (debug helper) — no compiled class, so ruleName is empty and the
+ // emitted source still includes probe lines with an empty rule string.
final MALExprCodegen exprCodegen = new MALExprCodegen(fieldNames);
return exprCodegen.generateRunMethod(ast);
}
diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALCodegenHelper.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALCodegenHelper.java
index 44457575de..948f6a949b 100644
--- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALCodegenHelper.java
+++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALCodegenHelper.java
@@ -21,6 +21,7 @@ import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
+import org.apache.skywalking.oap.server.core.dsldebug.DSLDebugCodegenSwitch;
/**
* Static utility methods and constants shared across MAL code generation classes.
@@ -80,6 +81,55 @@ final class MALCodegenHelper {
static final String RUNTIME_HELPER_FQCN =
"org.apache.skywalking.oap.meter.analyzer.v2.compiler.rt.MalRuntimeHelper";
+ /** FQCN of the per-rule capture binding the generated rule class carries. */
+ static final String GATE_HOLDER_FQCN =
+ "org.apache.skywalking.oap.server.core.dsldebug.GateHolder";
+
+ /** FQCN of the MAL probe class generated bytecode forwards to. */
+ static final String MAL_DEBUG_FQCN =
+ "org.apache.skywalking.oap.meter.analyzer.v2.dsldebug.MALDebug";
+
+ /**
+ * Emits a gate-guarded {@code MALDebug.captureStage} call site at the current position
+ * in the generated {@code run()} body. The guard reads the instance field
+ * {@code debug.isGateOn()} with one volatile load — {@code false} on the idle path and JIT
+ * eliminates the entire {@code if} after warm-up.
+ *
+ * <p>MAL probes carry the verbatim DSL fragment ({@code sourceText}) as the
+ * accurate locator; no {@code sourceLine} parameter — see {@code MALDebug}'s
+ * Javadoc for the rationale.
+ *
+ * @param sb buffer the emitted source line is appended to
+ * @param ruleName the rule's name — Java-string-escaped before emission
+ * @param sourceText verbatim DSL fragment for this stage — Java-string-escaped
+ * @param var the local variable holding the {@code SampleFamily} after the stage
+ */
+ static void emitCaptureStage(final StringBuilder sb, final String ruleName,
+ final String sourceText, final String var) {
+ if (!DSLDebugCodegenSwitch.isInjectionEnabled()) {
+ return;
+ }
+ sb.append(" if (this.debug.isGateOn()) ").append(MAL_DEBUG_FQCN)
+ .append(".captureStage(this.debug, \"").append(escapeJava(ruleName))
+ .append("\", \"").append(escapeJava(sourceText))
+ .append("\", ").append(var).append(");\n");
+ }
+
+ /**
+ * Emits a gate-guarded {@code MALDebug.captureInput} call site immediately after a
+ * metric reference is read from the input {@code samples} map.
+ */
+ static void emitCaptureInput(final StringBuilder sb, final String ruleName,
+ final String metricRef, final String var) {
+ if (!DSLDebugCodegenSwitch.isInjectionEnabled()) {
+ return;
+ }
+ sb.append(" if (this.debug.isGateOn()) ").append(MAL_DEBUG_FQCN)
+ .append(".captureInput(this.debug, \"").append(escapeJava(ruleName))
+ .append("\", \"").append(escapeJava(metricRef))
+ .append("\", ").append(var).append(");\n");
+ }
+
// ---- Method classification sets ----
/**
diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALExprCodegen.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALExprCodegen.java
index 49f67d8d2e..79f0ef0a78 100644
--- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALExprCodegen.java
+++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALExprCodegen.java
@@ -48,15 +48,35 @@ final class MALExprCodegen {
private final List<String> closureFieldNames;
private final MALMethodChainCodegen chainCodegen;
+ private final String ruleName;
private int closureFieldIndex;
private int varCounter;
private final Set<String> declaredVars = new HashSet<>();
MALExprCodegen(final List<String> closureFieldNames) {
+ this(closureFieldNames, "");
+ }
+
+ /**
+ * @param closureFieldNames pre-computed companion class field names
+ * @param ruleName rule identifier emitted as the first argument of every
+ * {@code MALDebug.captureXxx(...)} probe call site. Empty
+ * string when the codegen is being driven by a debug-only
+ * surface ({@link MALClassGenerator#generateSource}) where
+ * no class is actually compiled — the source view still
+ * includes the probe lines so debugging that path renders
+ * correctly.
+ */
+ MALExprCodegen(final List<String> closureFieldNames, final String ruleName) {
this.closureFieldNames = closureFieldNames;
+ this.ruleName = ruleName == null ? "" : ruleName;
this.chainCodegen = new MALMethodChainCodegen(this);
}
+ String getRuleName() {
+ return ruleName;
+ }
+
int getVarCount() {
return varCounter;
}
@@ -175,6 +195,9 @@ final class MALExprCodegen {
.append(") samples.getOrDefault(\"")
.append(MALCodegenHelper.escapeJava(expr.getMetricName()))
.append("\", ").append(SF).append(".EMPTY));\n");
+ // First read of an input metric — capture the SampleFamily as it enters the chain.
+ // The metric name itself is the sourceText: the UI shows "input: <metric>".
+ MALCodegenHelper.emitCaptureInput(sb, ruleName, expr.getMetricName(), var);
chainCodegen.emitChainStatements(sb, var, expr.getMethodChain());
return var;
}
@@ -215,6 +238,19 @@ final class MALExprCodegen {
final MALExpressionModel.Expr left = expr.getLeft();
final MALExpressionModel.Expr right = expr.getRight();
final MALExpressionModel.ArithmeticOp op = expr.getOp();
+ final String resultVar = emitBinaryExprBody(sb, left, right, op);
+ // Treat each binary op as one stage so the UI sees the value after the operator
+ // in the same waterfall column as named chain stages. The operator's method name
+ // (plus, minus, multiply, divide) is the verbatim DSL fragment.
+ MALCodegenHelper.emitCaptureStage(sb, ruleName,
+ MALCodegenHelper.opMethodName(op), resultVar);
+ return resultVar;
+ }
+
+ private String emitBinaryExprBody(final StringBuilder sb,
+ final MALExpressionModel.Expr left,
+ final MALExpressionModel.Expr right,
+ final MALExpressionModel.ArithmeticOp op) {
final boolean leftIsScalar = isScalar(left);
final boolean rightIsScalar = isScalar(right);
@@ -282,6 +318,7 @@ final class MALExprCodegen {
final String var = emitExpr(sb, expr.getOperand());
sb.append(" ").append(var).append(" = ")
.append(var).append(".negative();\n");
+ MALCodegenHelper.emitCaptureStage(sb, ruleName, "negative", var);
return var;
}
diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALExpressionModel.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALExpressionModel.java
index 6ab17dc2a2..06ca2b1468 100644
--- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALExpressionModel.java
+++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALExpressionModel.java
@@ -140,16 +140,51 @@ public final class MALExpressionModel {
private final String namespace;
private final String name;
private final List<Argument> arguments;
+ /**
+ * 1-based line number of this call in the source MAL expression. Threaded
+ * from ANTLR's {@code ctx.getStart().getLine()} at parse time so debug
+ * captures can point UI / CLI clients at the exact source line that
+ * produced the SampleFamily snapshot.
+ * <p>{@code 0} means "unknown / synthetic" — synthesised method calls
+ * (e.g. unit-test helpers) skip line tracking. Probe emission sites
+ * fall back to {@code 0} gracefully; the recorder is free to render
+ * "unknown line" in the UI.
+ */
+ private final int sourceLine;
+ /**
+ * Verbatim slice of the original MAL expression for this call —
+ * pulled from the ANTLR input stream via the call's start/stop
+ * indices, so whitespace and operator/identifier spelling match the
+ * source byte-for-byte. Used as the {@code sourceText} of dsl-debugging
+ * captures so the UI can match a captured stage record against the
+ * original {@code .yaml}'s {@code exp:} body without any
+ * reconstruction. {@code ""} for synthesised calls (unit-test helpers)
+ * — the probe path tolerates an empty source-text gracefully.
+ */
+ private final String sourceText;
public MethodCall(final String name, final List<Argument> arguments) {
- this(null, name, arguments);
+ this(null, name, arguments, 0, "");
}
public MethodCall(final String namespace, final String name,
final List<Argument> arguments) {
+ this(namespace, name, arguments, 0, "");
+ }
+
+ public MethodCall(final String namespace, final String name,
+ final List<Argument> arguments, final int sourceLine) {
+ this(namespace, name, arguments, sourceLine, "");
+ }
+
+ public MethodCall(final String namespace, final String name,
+ final List<Argument> arguments, final int sourceLine,
+ final String sourceText) {
this.namespace = namespace;
this.name = name;
this.arguments = Collections.unmodifiableList(arguments);
+ this.sourceLine = sourceLine;
+ this.sourceText = sourceText == null ? "" : sourceText;
}
public boolean isExtension() {
diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALMethodChainCodegen.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALMethodChainCodegen.java
index c428e99b90..d707c0a49e 100644
--- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALMethodChainCodegen.java
+++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALMethodChainCodegen.java
@@ -17,7 +17,9 @@
package org.apache.skywalking.oap.meter.analyzer.v2.compiler;
+import java.util.LinkedList;
import java.util.List;
+import java.util.Queue;
import org.apache.skywalking.oap.meter.analyzer.v2.compiler.rt.MalExtensionRegistry;
/**
@@ -44,7 +46,7 @@ final class MALMethodChainCodegen {
* Queue of pre-computed temp variable names for complex expression arguments.
* Filled before each method call, consumed by {@link #generateArgument} in order.
*/
- private java.util.Queue<String> preComputedArgs;
+ private Queue<String> preComputedArgs;
MALMethodChainCodegen(final MALExprCodegen exprCodegen) {
this.exprCodegen = exprCodegen;
@@ -73,9 +75,18 @@ final class MALMethodChainCodegen {
void emitChainStatements(final StringBuilder sb,
final String var,
final List<MALExpressionModel.MethodCall> chain) {
+ final String rule = exprCodegen.getRuleName();
for (final MALExpressionModel.MethodCall mc : chain) {
+ // Verbatim ANTLR slice of this chain segment, e.g.
+ // "sum(['service_name', 'step'])" or "tagEqual('region', 'us-east-1')".
+ // Captured directly from the input stream at parse time so the UI
+ // can match it byte-for-byte against the original .yaml's exp:
+ // body. Empty for synthesised calls (unit-test helpers).
+ final String stageText = mc.getSourceText();
+
if (mc.isExtension()) {
emitExtensionCall(sb, var, mc);
+ MALCodegenHelper.emitCaptureStage(sb, rule, stageText, var);
continue;
}
// Pre-compute complex expression arguments to temp variables
@@ -86,6 +97,8 @@ final class MALMethodChainCodegen {
emitBuiltinArgs(sb, var, mc);
sb.append(");\n");
preComputedArgs = null;
+ // Capture the SampleFamily after this chain stage runs.
+ MALCodegenHelper.emitCaptureStage(sb, rule, stageText, var);
}
}
@@ -201,9 +214,9 @@ final class MALMethodChainCodegen {
} else {
sb.append((int) raw);
}
- } else if (java.util.List.class.isAssignableFrom(expectedType)) {
+ } else if (List.class.isAssignableFrom(expectedType)) {
if (arg instanceof MALExpressionModel.StringListArgument) {
- final java.util.List<String> values =
+ final List<String> values =
((MALExpressionModel.StringListArgument) arg).getValues();
sb.append("java.util.Arrays.asList(new String[]{");
for (int i = 0; i < values.size(); i++) {
@@ -360,10 +373,10 @@ final class MALMethodChainCodegen {
* Scans arguments for complex expressions and pre-computes them to temp variables.
* Returns a queue of temp names consumed by {@link #generateArgument} in order.
*/
- private java.util.Queue<String> preComputeComplexArgs(
+ private Queue<String> preComputeComplexArgs(
final StringBuilder sb,
final List<MALExpressionModel.Argument> args) {
- final java.util.Queue<String> temps = new java.util.LinkedList<>();
+ final Queue<String> temps = new LinkedList<>();
for (final MALExpressionModel.Argument arg : args) {
if (!(arg instanceof MALExpressionModel.ExprArgument)) {
continue;
diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALScriptParser.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALScriptParser.java
index fce9bc2044..e37f285456 100644
--- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALScriptParser.java
+++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALScriptParser.java
@@ -23,11 +23,13 @@ import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.antlr.v4.runtime.BaseErrorListener;
+import org.antlr.v4.runtime.ParserRuleContext;
import org.antlr.v4.runtime.tree.TerminalNode;
import org.antlr.v4.runtime.CharStreams;
import org.antlr.v4.runtime.CommonTokenStream;
import org.antlr.v4.runtime.RecognitionException;
import org.antlr.v4.runtime.Recognizer;
+import org.antlr.v4.runtime.misc.Interval;
import org.apache.skywalking.mal.rt.grammar.MALLexer;
import org.apache.skywalking.mal.rt.grammar.MALParser;
import org.apache.skywalking.mal.rt.grammar.MALParserBaseVisitor;
@@ -261,7 +263,24 @@ public final class MALScriptParser {
}
final List<Argument> args = ctx.argumentList() != null
? visitArgList(ctx.argumentList()) : Collections.emptyList();
- return new MethodCall(namespace, name, args);
+ // Verbatim ANTLR slice of the call (e.g. "sum(['service_name', 'step'])").
+ // Pulled from the input stream so whitespace and identifier spelling are
+ // exactly what the operator wrote. Used as the captured stage's sourceText.
+ return new MethodCall(namespace, name, args,
+ ctx.getStart().getLine(), rawTextOf(ctx));
+ }
+
+ /**
+ * Verbatim source slice of an ANTLR rule context — pulls from the
+ * original input stream rather than re-concatenating tokens, so
+ * whitespace inside the call is preserved.
+ */
+ private static String rawTextOf(final ParserRuleContext ctx) {
+ if (ctx.getStart() == null || ctx.getStop() == null) {
+ return "";
+ }
+ return ctx.getStart().getInputStream().getText(
+ Interval.of(ctx.getStart().getStartIndex(), ctx.getStop().getStopIndex()));
}
private List<Argument> visitArgList(final MALParser.ArgumentListContext ctx) {
diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/DSL.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/DSL.java
index 3fc72f4701..65a252b5eb 100644
--- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/DSL.java
+++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/DSL.java
@@ -103,9 +103,14 @@ public final class DSL {
// yamlSource state that the shared GENERATOR carries between calls.
final MALClassGenerator perFile = new MALClassGenerator(pool, targetClassLoader);
perFile.setYamlSource(yamlSource);
+ // The verbatim MAL expression text is the rule's "content" — threaded
+ // into the GateHolder so dsl-debugging records carry the rule source
+ // inline. Same string ANTLR parsed; no extra plumbing.
+ perFile.setContent(expression);
malExpr = perFile.compile(metricName, expression);
} else {
GENERATOR.setYamlSource(yamlSource);
+ GENERATOR.setContent(expression);
malExpr = GENERATOR.compile(metricName, expression);
}
return new Expression(metricName, expression, malExpr);
diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/Expression.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/Expression.java
index f7b012a963..2589cc52a1 100644
--- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/Expression.java
+++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/Expression.java
@@ -20,6 +20,7 @@ package org.apache.skywalking.oap.meter.analyzer.v2.dsl;
import java.util.Map;
import lombok.ToString;
import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.dsldebug.GateHolder;
/**
* Wraps a compiled {@link MalExpression} with runtime state management.
@@ -51,6 +52,16 @@ public class Expression {
return expression.getClass().getName();
}
+ /**
+ * Returns the per-rule capture binding the compiled rule carries. The
+ * {@code Analyzer.doAnalysis} hand-written probes read it once per ingest
+ * pass and then gate-guard {@code MALDebug.captureXxx(...)} calls on
+ * the holder's volatile {@code gate} field.
+ */
+ public GateHolder debugHolder() {
+ return expression.debugHolder();
+ }
+
/**
* Returns compile-time metadata extracted from the expression AST.
*/
diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/FilterExpression.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/FilterExpression.java
index 49a6ef2d3e..8a7838b0de 100644
--- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/FilterExpression.java
+++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/FilterExpression.java
@@ -21,6 +21,7 @@ import java.util.HashMap;
import java.util.Map;
import java.util.Objects;
import javassist.ClassPool;
+import lombok.Getter;
import lombok.ToString;
import lombok.extern.slf4j.Slf4j;
import org.apache.skywalking.oap.meter.analyzer.v2.compiler.MALClassGenerator;
@@ -34,6 +35,7 @@ import org.apache.skywalking.oap.meter.analyzer.v2.compiler.MALClassGenerator;
public class FilterExpression {
private static final MALClassGenerator GENERATOR = new MALClassGenerator();
+ @Getter
private final String literal;
private final MalFilter malFilter;
diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/MalExpression.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/MalExpression.java
index 32a24dd766..d6c2fc11e4 100644
--- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/MalExpression.java
+++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/MalExpression.java
@@ -19,6 +19,7 @@
package org.apache.skywalking.oap.meter.analyzer.v2.dsl;
import java.util.Map;
+import org.apache.skywalking.oap.server.core.dsldebug.GateHolder;
/**
* Each compiled MAL expression implements this interface.
@@ -31,4 +32,18 @@ public interface MalExpression {
* sample names, scope type, aggregation labels, downsampling, etc.
*/
ExpressionMetadata metadata();
+
+ /**
+ * Returns the per-rule capture binding the generated bytecode reads on
+ * every probe call site. Codegen emits the {@code public final GateHolder
+ * DEBUG} field + the override only when
+ * {@link org.apache.skywalking.oap.server.core.dsldebug.DSLDebugCodegenSwitch#isInjectionEnabled()
+ * dsl-debugging is enabled at boot}. When injection is off, this default
+ * returns {@code null} and the session-install path treats that as
+ * "no live holder for this rule" (i.e. the session is not capturable on
+ * this node).
+ */
+ default GateHolder debugHolder() {
+ return null;
+ }
}
diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/SampleFamily.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/SampleFamily.java
index 77501a45b1..c2c2ee07c6 100644
--- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/SampleFamily.java
+++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/SampleFamily.java
@@ -57,6 +57,10 @@ import com.google.common.base.Preconditions;
import com.google.common.base.Strings;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.Maps;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonNull;
+import com.google.gson.JsonObject;
import io.vavr.Function2;
import io.vavr.Function3;
import org.apache.skywalking.oap.meter.analyzer.v2.dsl.SampleFamilyFunctions.DecorateFunction;
@@ -119,6 +123,53 @@ public class SampleFamily {
return this;
}
+ /**
+ * Field-by-field JSON payload for the dsl-debugging capture. Emits
+ * the complete sample list — name, labels, value, timestamp for
+ * every sample in this family. With kept-only capture (rejected
+ * executions are discarded before they reach the recorder), the
+ * family that lands here is by definition relevant to the operator,
+ * so the full payload is appropriate for a debug API.
+ *
+ * <p>Built via Gson's {@link JsonObject} —
+ * library-handled escaping, explicit field selection (no Gson
+ * reflection-based serialization).
+ */
+ public String toJson() {
+ final JsonObject obj = new JsonObject();
+ if (this == EMPTY || samples == null || samples.length == 0) {
+ obj.addProperty("empty", true);
+ obj.addProperty("samples", 0);
+ obj.add("items", new JsonArray());
+ return obj.toString();
+ }
+ obj.addProperty("samples", samples.length);
+ final JsonArray items = new JsonArray();
+ for (final Sample s : samples) {
+ items.add(sampleToJson(s));
+ }
+ obj.add("items", items);
+ return obj.toString();
+ }
+
+ private static JsonElement sampleToJson(final Sample s) {
+ if (s == null) {
+ return JsonNull.INSTANCE;
+ }
+ final JsonObject item = new JsonObject();
+ item.addProperty("name", s.getName());
+ final JsonObject labels = new JsonObject();
+ if (s.getLabels() != null) {
+ for (final Map.Entry<String, String> e : s.getLabels().entrySet()) {
+ labels.addProperty(e.getKey(), e.getValue());
+ }
+ }
+ item.add("labels", labels);
+ item.addProperty("value", s.getValue());
+ item.addProperty("timestamp", s.getTimestamp());
+ return item;
+ }
+
/**
* Following operations are used in DSL
*/
diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsldebug/MALDebug.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsldebug/MALDebug.java
new file mode 100644
index 0000000000..4e8fe4f862
--- /dev/null
+++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsldebug/MALDebug.java
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.meter.analyzer.v2.dsldebug;
+
+import java.util.Map;
+import org.apache.skywalking.oap.meter.analyzer.v2.dsl.SampleFamily;
+import org.apache.skywalking.oap.server.core.analysis.meter.MeterEntity;
+import org.apache.skywalking.oap.server.core.analysis.meter.function.AcceptableValue;
+import org.apache.skywalking.oap.server.core.dsldebug.DebugRecorder;
+import org.apache.skywalking.oap.server.core.dsldebug.GateHolder;
+
+/**
+ * Probe surface generated MAL bytecode calls into. Each method takes the
+ * per-rule {@link GateHolder} as its first argument and fans the capture
+ * out to every recorder currently bound on that holder.
+ *
+ * <h2>Probe payload — verbatim DSL fragment, no line numbers</h2>
+ * MAL probes carry the verbatim DSL fragment ({@code "sum(['svc'])"},
+ * {@code "tagEqual('region', 'us-east-1')"}, {@code "myext::scale(2.0)"})
+ * as the {@code sourceText} argument. There is no separate line number —
+ * MAL stages don't map cleanly to a single AST line (binary ops, scope
+ * suffixes, and chain stages all interleave), so the verbatim text is the
+ * accurate locator. The recorder serialises {@code sourceText} as-is and
+ * the UI renders it directly.
+ *
+ * <h2>Workflow</h2>
+ * <pre>
+ * Generated rule's run() body:
+ * if (debug.gate) MALDebug.captureInput(debug, ruleName, "metric_ref", _v);
+ * ...stage...
+ * if (debug.gate) MALDebug.captureStage(debug, ruleName, "stageText", _v);
+ *
+ * Analyzer.doAnalysis (hand-written):
+ * if (debug.gate) MALDebug.captureFilter(debug, ruleName, filterText, family, kept);
+ * ...meterSamples.forEach...
+ * if (debug.gate) MALDebug.captureMeterBuild(debug, ruleName, entity, name, type, v);
+ * if (debug.gate) MALDebug.captureMeterEmit(debug, ruleName, entity, name, v, ts);
+ * send(...)
+ * </pre>
+ */
+public final class MALDebug {
+
+ private MALDebug() {
+ }
+
+ /** Functional dispatcher used by every probe to forward to one recorder. */
+ @FunctionalInterface
+ private interface Dispatch {
+ void to(MALDebugRecorder recorder);
+ }
+
+ /**
+ * Marks the start of one analysis pass for this rule. The recorder
+ * uses it as the per-execution boundary so all probes between begin
+ * and end land in one captured record. Fired at the top of
+ * {@code Analyzer.doAnalysis} for this rule.
+ */
+ public static void captureBeginAnalysis(final GateHolder holder, final String rule) {
+ fanOut(holder, r -> r.appendBeginAnalysis(rule));
+ }
+
+ /**
+ * Marks the end of one analysis pass. Fired in a finally block in
+ * {@code Analyzer.doAnalysis} so a thrown exception still publishes
+ * the partial record.
+ */
+ public static void captureEndAnalysis(final GateHolder holder, final String rule) {
+ fanOut(holder, r -> r.appendEndAnalysis(rule));
+ }
+
+ public static void captureInput(final GateHolder holder, final String rule,
+ final String metricRef, final SampleFamily family) {
+ fanOut(holder, r -> r.appendInput(rule, metricRef, family));
+ }
+
+ /**
+ * File-level filter capture. The {@code surviving} map carries every
+ * SampleFamily that passed the file-level filter — keyed by metric
+ * name, one entry per metric reference the rule's expression reads
+ * (so a rule that combines {@code metric_a + metric_b + metric_c}
+ * shows up to three entries; metrics whose samples were entirely
+ * filtered out are absent). {@code kept} is the per-rule survival
+ * verdict — {@code true} when at least one family survived, {@code
+ * false} when the file-level filter short-circuits the rule. Probes
+ * on both branches so the UI can show "filter dropped this batch".
+ */
+ public static void captureFilter(final GateHolder holder, final String rule,
+ final String filterText,
+ final Map<String, SampleFamily> surviving,
+ final boolean kept) {
+ fanOut(holder, r -> r.appendFilter(rule, filterText, surviving, kept));
+ }
+
+ public static void captureStage(final GateHolder holder, final String rule,
+ final String sourceText, final SampleFamily family) {
+ fanOut(holder, r -> r.appendStage(rule, sourceText, family));
+ }
+
+ public static void captureDownsample(final GateHolder holder, final String rule,
+ final String function, final String origin,
+ final SampleFamily family) {
+ fanOut(holder, r -> r.appendDownsample(rule, function, origin, family));
+ }
+
+ public static void captureMeterEmit(final GateHolder holder, final String rule,
+ final MeterEntity entity, final String metricName,
+ final AcceptableValue<?> value, final long timeBucket) {
+ fanOut(holder, r -> r.appendMeterEmit(rule, entity, metricName, value, timeBucket));
+ }
+
+ private static void fanOut(final GateHolder holder, final Dispatch dispatch) {
+ final DebugRecorder[] snapshot = holder.getRecorders();
+ for (int i = 0; i < snapshot.length; i++) {
+ final DebugRecorder recorder = snapshot[i];
+ if (recorder.isCaptured()) {
+ continue;
+ }
+ dispatch.to((MALDebugRecorder) recorder);
+ }
+ }
+}
diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsldebug/MALDebugRecorder.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsldebug/MALDebugRecorder.java
new file mode 100644
index 0000000000..e77d293e1c
--- /dev/null
+++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsldebug/MALDebugRecorder.java
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.meter.analyzer.v2.dsldebug;
+
+import java.util.Map;
+import org.apache.skywalking.oap.meter.analyzer.v2.dsl.SampleFamily;
+import org.apache.skywalking.oap.server.core.analysis.meter.MeterEntity;
+import org.apache.skywalking.oap.server.core.analysis.meter.function.AcceptableValue;
+import org.apache.skywalking.oap.server.core.dsldebug.DebugRecorder;
+
+/**
+ * Per-session recorder for MAL rules. The probe class {@link MALDebug} fans
+ * every call out to every recorder bound on the rule's {@code GateHolder},
+ * downcasting to this interface inside the loop.
+ *
+ * <p>Probe payloads carry verbatim DSL fragments as the {@code sourceText}
+ * — MAL doesn't have a clean per-line mapping (binary ops / scope suffixes
+ * span multiple AST nodes), so the verbatim source text is the accurate
+ * locator the UI renders. No {@code sourceLine} field.
+ */
+public interface MALDebugRecorder extends DebugRecorder {
+
+ /**
+ * Marks the start of one analysis pass — fired at the top of
+ * {@code Analyzer.doAnalysis} for this rule. The recorder uses it
+ * as the per-execution boundary: any leftover in-flight execution
+ * from a prior pass that didn't reach a terminal probe (an empty
+ * meterSamples that fired no meterEmit) is flushed here so the
+ * next pass starts on a clean slate.
+ *
+ * <p>One MAL execution = one analysis pass = the
+ * {@code Analyzer.doAnalysis} call for this rule on one OTLP
+ * push. All probes between
+ * {@link #appendBeginAnalysis}/{@link #appendEndAnalysis} land in
+ * one {@link org.apache.skywalking.oap.server.admin.dsl.debugging.session.ExecutionRecord}.
+ * Default no-op so recorders that don't bucket by analysis pass
+ * (e.g. test fixtures) don't have to implement it.
+ */
+ default void appendBeginAnalysis(String rule) {
+ }
+
+ /**
+ * Marks the end of one analysis pass — fired in a finally block in
+ * {@code Analyzer.doAnalysis}. The recorder publishes the current
+ * in-flight execution here. Default no-op.
+ */
+ default void appendEndAnalysis(String rule) {
+ }
+
+ /** First read of an input metric reference at the top of run(). */
+ void appendInput(String rule, String metricRef, SampleFamily family);
+
+ /**
+ * File-level filter probe. {@code surviving} carries every
+ * SampleFamily that passed the filter, keyed by metric name — so
+ * a rule combining {@code metric_a + metric_b + metric_c} shows up
+ * to three entries (metrics filtered to EMPTY are absent). {@code
+ * kept = true} when at least one family survived, {@code false}
+ * when the rule short-circuits post-filter.
+ */
+ void appendFilter(String rule, String filterText, Map<String, SampleFamily> surviving, boolean kept);
+
+ /** Per-stage capture (chain method, binary op, unary neg, etc.). */
+ void appendStage(String rule, String sourceText, SampleFamily family);
+
+ /** Downsample stage (rate / irate / increase / etc.). */
+ void appendDownsample(String rule, String function, String origin, SampleFamily family);
+
+ /**
+ * Captures the {@code (value, timeBucket)} pair the ingest path is about to push —
+ * the L1-bound boundary that ends MAL's debugger scope.
+ *
+ * <p>Note: there is no separate {@code appendMeterBuild} probe. In MAL the typed
+ * {@code AcceptableValue} is constructed and {@code accept()}-populated in one
+ * step, immediately followed by emit. The pre-emit and post-emit states are
+ * identical, so capturing them as separate samples would emit duplicate payloads
+ * (unlike OAL where {@code appendBuild} fires before the entry function runs).
+ */
+ void appendMeterEmit(String rule, MeterEntity entity, String metricName,
+ AcceptableValue<?> value, long timeBucket);
+}
diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsldebug/MalStaticBindingHook.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsldebug/MalStaticBindingHook.java
new file mode 100644
index 0000000000..44f5042de7
--- /dev/null
+++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsldebug/MalStaticBindingHook.java
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.meter.analyzer.v2.dsldebug;
+
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.skywalking.oap.meter.analyzer.v2.Analyzer;
+import org.apache.skywalking.oap.meter.analyzer.v2.MetricConvert;
+import org.apache.skywalking.oap.server.core.dsldebug.GateHolder;
+
+/**
+ * Process-wide hook that publishes static-loader MAL rule bindings into the
+ * dsl-debugging session registry. Avoids plumbing a {@code MALHolderRegistry}
+ * reference through every receiver-plugin's static-rule loader: each loader
+ * builds its {@link MetricConvert} as before, then calls
+ * {@link #publish(String, String, MetricConvert)} once. The hook is no-op
+ * until the dsl-debugging module installs a real sink in its provider's
+ * {@code start()} phase.
+ *
+ * <h2>Workflow</h2>
+ * <pre>
+ * dsl-debugging start() ─► MalStaticBindingHook.install(sink)
+ *
+ * receiver-plugin boot loop:
+ * MetricConvert convert = new MetricConvert(rule, meterSystem);
+ * MalStaticBindingHook.publish("otel-rules", rule.getName(), convert);
+ * │
+ * └─ for each analyzer in convert:
+ * sink.publish(catalog, name, metricName, holder)
+ * (sink wraps holder into MALHolderRegistry.register)
+ * </pre>
+ *
+ * <p>Why a static hook and not an SPI: MAL static loaders run before any
+ * module-manager service is available in some test topologies, and walking
+ * {@code ServiceLoader} on every static rule load is wasteful when the
+ * binding cost is one method call. The hook is process-scoped, set once,
+ * and never overwritten in a normal OAP boot. Tests that don't enable
+ * dsl-debugging see the no-op default and pay nothing.
+ */
+public final class MalStaticBindingHook {
+
+ /** Sink the hook forwards every bound holder to. */
+ @FunctionalInterface
+ public interface Sink {
+ void publish(String catalog, String name, String metricName, GateHolder holder);
+ }
+
+ private static final Sink NOOP = (catalog, name, metricName, holder) -> {
+ };
+
+ /**
+ * Active sink reference. AtomicReference (rather than a plain volatile
+ * field) keeps the field name conventionally lowercase by treating it as
+ * a final container holding mutable state — matches checkstyle's
+ * static-naming rule without forcing the SINK identifier on the API.
+ */
+ private static final AtomicReference<Sink> SINK = new AtomicReference<>(NOOP);
+
+ private MalStaticBindingHook() {
+ }
+
+ /**
+ * Install the active sink. Called once by the dsl-debugging module
+ * provider's {@code prepare()}. A second install replaces the prior
+ * sink — useful in tests; harmless in production where only
+ * dsl-debugging installs one sink per OAP process.
+ */
+ public static void install(final Sink s) {
+ SINK.set(s == null ? NOOP : s);
+ }
+
+ /** Reset to the no-op sink. Used by tests after install/teardown. */
+ public static void reset() {
+ SINK.set(NOOP);
+ }
+
+ /**
+ * Walk the supplied {@link MetricConvert}'s analyzers and forward each
+ * {@code (catalog, name, metricName, debugHolder)} tuple to the active
+ * sink. Convert is read-only (we do not mutate). No-op when the convert
+ * is null or carries no analyzers.
+ */
+ public static void publish(final String catalog, final String name,
+ final MetricConvert convert) {
+ if (convert == null || catalog == null || name == null) {
+ return;
+ }
+ final Sink s = SINK.get();
+ for (final Analyzer analyzer : convert.getAnalyzers()) {
+ if (analyzer.getExpression() == null) {
+ continue;
+ }
+ s.publish(catalog, name, analyzer.getMetricName(),
+ analyzer.getExpression().debugHolder());
+ }
+ }
+}
diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALClassGeneratorTest.java b/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALClassGeneratorTest.java
index f3e9c9287a..f6594c66e8 100644
--- a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALClassGeneratorTest.java
+++ b/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/v2/compiler/MALClassGeneratorTest.java
@@ -41,7 +41,18 @@ class MALClassGeneratorTest {
@BeforeEach
void setUp() {
+ // Mirror the production toggle so a developer can rebuild with
+ // SW_DSL_DEBUGGING_INJECTION_ENABLED=true and inspect the
+ // debug-probe-injected generated classes (gate field +
+ // probe call sites) in target/mal-generated-classes/.
+ if ("true".equalsIgnoreCase(System.getenv("SW_DSL_DEBUGGING_INJECTION_ENABLED"))) {
+ org.apache.skywalking.oap.server.core.dsldebug.DSLDebugCodegenSwitch.enableInjection();
+ } else {
+ org.apache.skywalking.oap.server.core.dsldebug.DSLDebugCodegenSwitch.resetInjection();
+ }
generator = new MALClassGenerator(new ClassPool(true));
+ generator.setClassOutputDir(
+ org.apache.skywalking.oap.server.testing.dsl.DslClassOutput.unitTestDir("mal"));
}
// ==================== Basic compilation tests ====================
diff --git a/oap-server/oal-rt/CLAUDE.md b/oap-server/oal-rt/CLAUDE.md
index d0c476589e..67bf080437 100644
--- a/oap-server/oal-rt/CLAUDE.md
+++ b/oap-server/oal-rt/CLAUDE.md
@@ -88,14 +88,22 @@ model.getMetricsClassName(); // e.g., "LongAvgMetrics"
## Debug Output
-When `SW_DYNAMIC_CLASS_ENGINE_DEBUG=true` environment variable is set, generated `.class` files are written to disk for inspection:
+When `SW_DYNAMIC_CLASS_ENGINE_DEBUG=true` environment variable is set, generated `.class` files are written to disk for inspection. Each `.class` is paired with a `<ClassName>.java` sidecar — the verbatim Java source the codegen fed Javassist:
```
{skywalking}/oal-rt/
-├── metrics/ - Generated metrics .class files
-└── dispatcher/ - Generated dispatcher .class files
+├── metrics/
+│ ├── ServiceRespTimeMetrics.class
+│ └── ServiceRespTimeMetrics.java ← Javassist compile input (synthetic)
+└── dispatcher/
+ ├── ServiceRelationDispatcher.class
+ └── ServiceRelationDispatcher.java
```
+The `.java` sidecar exists so IDE source-attach renders the actual codegen input directly without relying on FernFlower / a decompiler. Javassist-emitted bytecode often confuses decompilers (no `goto` consolidation, slot reuse with mixed types, debug-injected `if (gate.isGateOn()) { ... }` chains), and FernFlower frequently bails to "compiled code" stubs. The source-attach path always works and shows the EXACT code Javassist compiled — gate fields, probe call sites, the lot.
+
+When `SW_DSL_DEBUGGING_INJECTION_ENABLED=true` is also set, the codegen flips to the with-debug template (`doMetricsWithDebug.ftl`), which adds the per-metric `GateHolder debug_<metric>` fields and `OALDebug.captureXxx(...)` probe sites to both the `.class` and the `.java` sidecar. The two env vars are independent: `SW_DYNAMIC_CLASS_ENGINE_DEBUG` controls disk dump; `SW_DSL_DEBUGGING_INJECTION_ENABLED` controls codegen branch.
+
This is useful for debugging code generation issues or comparing V1 vs V2 output.
## Runtime Integration
diff --git a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/OALEngineV2.java b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/OALEngineV2.java
index c321c80fcd..7e21a138c3 100644
--- a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/OALEngineV2.java
+++ b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/OALEngineV2.java
@@ -93,7 +93,10 @@ public class OALEngineV2 implements OALEngine {
// Prepare temp folder for generated classes
classGeneratorV2.prepareRTTempFolder();
- // Load OAL script, parse, and generate classes with proper resource management
+ // Load OAL script, parse, and generate classes. Per-metric content for
+ // dsl-debugging is captured by the parser as an ANTLR Interval slice over
+ // each metric's full statement (see OALListenerV2.exitAggregationStatement)
+ // — no whole-file plumbing needed.
try (Reader reader = ResourceUtils.read(oalDefine.getConfigFile())) {
// Parse using V2 parser
OALScriptParserV2 v2Parser;
diff --git a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/generator/CodeGenModel.java b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/generator/CodeGenModel.java
index 080775b668..f877634191 100644
--- a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/generator/CodeGenModel.java
+++ b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/generator/CodeGenModel.java
@@ -114,6 +114,32 @@ public class CodeGenModel {
*/
private String functionName;
+ /**
+ * Verbatim OAL aggregation clause for debug-capture sourceText —
+ * e.g. {@code "cpm()"} or {@code "percentile2(10)"}. Lets the UI
+ * highlight the aggregation step in the original {@code .oal} source.
+ */
+ private String aggregationSourceText;
+
+ /**
+ * 1-based source line of this rule in the {@code .oal} file. Threaded
+ * from the parser's {@code SourceLocation} at enrichment time and
+ * spliced into every dsl-debugging probe call site so captured records
+ * carry an in-source pointer alongside the verbatim {@code sourceText}.
+ */
+ @Builder.Default
+ private int sourceLine = 0;
+
+ /**
+ * Verbatim text of this metric's full {@code .oal} statement —
+ * {@code "service_relation_server_cpm = from(ServiceRelation.*).filter(...).cpm();"}
+ * — captured from the ANTLR input stream at parse time. Threaded into
+ * the per-metric {@code GateHolder} so dsl-debugging records render the
+ * rule source inline. Empty for synthesised metrics.
+ */
+ @Builder.Default
+ private String metricSourceText = "";
+
/**
* Metrics function class name.
* Example: "LongAvgMetrics"
@@ -369,6 +395,14 @@ public class CodeGenModel {
private String left;
/** Right operand expression (literal or source getter). */
private String right;
+ /**
+ * Verbatim OAL filter clause for debug-capture sourceText —
+ * e.g. {@code "filter(detectPoint == DetectPoint.SERVER)"}. The UI
+ * uses this to highlight the filter clause in the original .oal
+ * source; the matcher class name on its own ({@code expressionObject})
+ * is generated and meaningless to operators.
+ */
+ private String sourceText;
}
/**
@@ -384,6 +418,14 @@ public class CodeGenModel {
private String sourceName;
/** Scope ID for this source type. */
private int sourceScopeId;
+ /**
+ * Verbatim {@code from(SourceName.attr)} or {@code from(SourceName.*)}
+ * slice — what the operator literally wrote in the .oal file.
+ * Threaded into the dsl-debugging capture so the source sample's
+ * {@code sourceText} matches the original source clause byte-for-byte
+ * (the matcher / scope-id are generated and meaningless to operators).
+ */
+ private String fromText;
}
/**
diff --git a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/generator/MetricDefinitionEnricher.java b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/generator/MetricDefinitionEnricher.java
index 12225419be..d427f662f6 100644
--- a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/generator/MetricDefinitionEnricher.java
+++ b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/generator/MetricDefinitionEnricher.java
@@ -131,8 +131,24 @@ public class MetricDefinitionEnricher {
.from(CodeGenModel.FromStmtV2.builder()
.sourceName(metric.getSource().getName())
.sourceScopeId(sourceScopeId)
+ // ANTLR Interval slice captured at parse time on the
+ // metricStatement context (see OALListenerV2.exitMetricStatement)
+ // — already byte-identical to what the operator wrote, no
+ // runtime reconstruction needed at the capture call site.
+ .fromText(metric.getFromText())
.build())
.functionName(metric.getAggregationFunction().getName())
+ // Verbatim ANTLR slice from the parser — e.g. "cpm()", "percentile2(10)".
+ // Captured byte-for-byte from the original .oal so the UI can match an
+ // aggregation capture record against the source.
+ .aggregationSourceText(metric.getAggregationFunction().getSourceText())
+ // 1-based source line in core.oal — threaded into every dsl-debugging
+ // probe call site so captured records carry an in-source pointer.
+ .sourceLine(metric.getLocation().getLine())
+ // Verbatim ANTLR slice of this metric's full statement — used as the
+ // per-metric GateHolder's content so dsl-debugging records carry the
+ // rule source inline.
+ .metricSourceText(metric.getSourceText())
.metricsClassName(metricsClassName)
.filters(CodeGenModel.FiltersV2.builder()
.filterExpressions(filterExpressions)
@@ -565,10 +581,15 @@ public class MetricDefinitionEnricher {
String matcherClass = getMatcherClassName(filter);
String left = buildFilterLeft(filter);
String right = buildFilterRight(filter);
+ // Verbatim ANTLR slice from the parser — e.g.
+ // "filter(detectPoint == DetectPoint.SERVER)". Used as the
+ // captured filter record's sourceText so the UI matches against
+ // the original .oal source byte-for-byte.
result.add(CodeGenModel.FilterExpressionV2.builder()
.expressionObject(matcherClass)
.left(left)
.right(right)
+ .sourceText(filter.getSourceText())
.build());
}
return result;
diff --git a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/generator/OALClassGeneratorV2.java b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/generator/OALClassGeneratorV2.java
index 25aefe9018..3313ca8288 100644
--- a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/generator/OALClassGeneratorV2.java
+++ b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/generator/OALClassGeneratorV2.java
@@ -25,6 +25,7 @@ import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.StringWriter;
+import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Locale;
@@ -50,6 +51,7 @@ import org.apache.commons.io.FileUtils;
import org.apache.skywalking.oap.server.core.WorkPath;
import org.apache.skywalking.oap.server.core.analysis.DisableRegister;
import org.apache.skywalking.oap.server.core.analysis.SourceDispatcher;
+import org.apache.skywalking.oap.server.core.dsldebug.DSLDebugCodegenSwitch;
import org.apache.skywalking.oap.server.core.analysis.Stream;
import org.apache.skywalking.oap.server.core.oal.rt.OALCompileException;
import org.apache.skywalking.oap.server.core.oal.rt.OALDefine;
@@ -73,6 +75,10 @@ public class OALClassGeneratorV2 {
private static final String METRICS_FUNCTION_PACKAGE = "org.apache.skywalking.oap.server.core.analysis.metrics.";
private static final String WITH_METADATA_INTERFACE = "org.apache.skywalking.oap.server.core.analysis.metrics.WithMetadata";
private static final String DISPATCHER_INTERFACE = "org.apache.skywalking.oap.server.core.analysis.SourceDispatcher";
+ private static final String DEBUG_HOLDER_PROVIDER_INTERFACE =
+ "org.apache.skywalking.oap.server.core.dsldebug.DebugHolderProvider";
+ private static final String GATE_HOLDER_CLASS =
+ "org.apache.skywalking.oap.server.core.dsldebug.GateHolder";
private static final String METRICS_STREAM_PROCESSOR = "org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor";
private static final String[] METRICS_CLASS_METHODS = {
"id",
@@ -100,6 +106,9 @@ public class OALClassGeneratorV2 {
private Configuration configuration;
private StorageBuilderFactory storageBuilderFactory;
private static String GENERATED_FILE_PATH;
+ // Per-metric content for dsl-debugging holders is read from each
+ // CodeGenModel's metricSourceText field (populated by the parser via
+ // ANTLR Interval slice), so no whole-file content is held here.
public OALClassGeneratorV2(OALDefine define) {
this(define, ClassPool.getDefault());
@@ -253,13 +262,16 @@ public class OALClassGeneratorV2 {
}
// Generate methods using V2 templates
+ final StringBuilder sourceMethods = new StringBuilder();
for (String method : METRICS_CLASS_METHODS) {
StringWriter methodEntity = new StringWriter();
try {
configuration.getTemplate("metrics/" + method + ".ftl").process(model, methodEntity);
- javassist.CtMethod m = CtNewMethod.make(methodEntity.toString(), metricsClass);
+ final String body = methodEntity.toString();
+ javassist.CtMethod m = CtNewMethod.make(body, metricsClass);
metricsClass.addMethod(m);
addLineNumberTable(m, 1);
+ sourceMethods.append(" ").append(body.replace("\n", "\n ")).append("\n\n");
} catch (Exception e) {
log.error("Can't generate method " + method + " for " + className + ".", e);
throw new OALCompileException(e.getMessage(), e);
@@ -291,10 +303,29 @@ public class OALClassGeneratorV2 {
log.debug("Generated V2 metrics class: " + metricsClass.getName());
writeGeneratedFile(metricsClass, "metrics");
+ writeGeneratedSourceFile(metricsClass, "metrics",
+ wrapMetricsClassSource(metricsClass, model, sourceMethods.toString()));
return targetClass;
}
+ /** Wraps generated method bodies + fields as a compilable class envelope. */
+ private String wrapMetricsClassSource(final CtClass ctClass, final CodeGenModel model, final String methods) {
+ final StringBuilder sb = new StringBuilder();
+ sb.append("package ").append(ctClass.getPackageName()).append(";\n\n");
+ sb.append("public class ").append(ctClass.getSimpleName())
+ .append(" extends ").append(model.getMetricsClassName())
+ .append(" implements ").append(WITH_METADATA_INTERFACE).append(" {\n\n");
+ for (final CodeGenModel.SourceFieldV2 field : model.getFieldsFromSource()) {
+ sb.append(" private ").append(field.getType().getName()).append(' ')
+ .append(field.getFieldName()).append(";\n");
+ }
+ sb.append('\n');
+ sb.append(methods);
+ sb.append("}\n");
+ return sb.toString();
+ }
+
/**
* Generate metrics builder class using V2 model and templates.
*/
@@ -357,6 +388,8 @@ public class OALClassGeneratorV2 {
try {
CtClass dispatcherInterface = classPool.get(DISPATCHER_INTERFACE);
dispatcherClass.addInterface(dispatcherInterface);
+ // DebugHolderProvider is added below only when dsl-debugging injection is
+ // enabled at boot — paired with the per-source `debug` field and accessors.
// Set generic signature
String sourceClassName = oalDefine.getSourcePackage() + dispatcherContext.getSourceName();
@@ -380,14 +413,103 @@ public class OALClassGeneratorV2 {
throw new OALCompileException(e.getMessage(), e);
}
- // Generate doMetrics methods for each metric
+ // OAL gate is PER-METRIC: one GateHolder per do<Metric>() in the dispatcher.
+ // Each metric's session sees only its own rule's pipeline — the other
+ // metrics on the same source dispatcher don't fire any probes because
+ // their gates stay off. Symmetric with MAL/LAL session granularity.
+ //
+ // Emitted only when dsl-debugging is enabled at boot. When off, the
+ // dispatcher does not implement DebugHolderProvider, the fields are
+ // absent, and doMetrics.ftl skips every probe call site — bytecode
+ // matches a build without SWIP-13.
+ if (DSLDebugCodegenSwitch.isInjectionEnabled()) {
+ final String holderFqcn = "org.apache.skywalking.oap.server.core.dsldebug.GateHolder";
+ try {
+ dispatcherClass.addInterface(classPool.get(DEBUG_HOLDER_PROVIDER_INTERFACE));
+
+ // One `debug_<oalRuleName>` GateHolder field per metric. The OAL
+ // rule's user-facing name is snake_case (e.g. service_relation_server_cpm)
+ // and is what operators type in `.oal` files — that's the lookup key
+ // we expose. Each holder carries its own metric's verbatim ANTLR slice
+ // as content (the single-line OAL statement) so dsl-debugging records
+ // render the exact rule source inline. The snake_case name is a valid
+ // Java identifier, so we splice it straight into the field name —
+ // `debug_service_relation_server_cpm`.
+ for (CodeGenModel metric : dispatcherContext.getMetrics()) {
+ final String perMetricContent = escapeJavaLiteral(metric.getMetricSourceText());
+ // Use the GateHolder.withMetadata factory so each holder is
+ // stamped with {ruleName, sourceLine} at instance-init time
+ // — the dsl-debugging records carry a structured per-rule
+ // envelope alongside the verbatim dsl source.
+ dispatcherClass.addField(javassist.CtField.make(
+ "public final " + holderFqcn + " debug_" + metric.getTableName()
+ + " = " + holderFqcn + ".withMetadata(\""
+ + perMetricContent + "\", \""
+ + escapeJavaLiteral(metric.getTableName()) + "\", "
+ + metric.getSourceLine() + ");",
+ dispatcherClass));
+ }
+
+ // debugHolder(String) returns the right field by OAL rule name.
+ // If/else chain — typical dispatcher has 2-10 metrics; binary
+ // search isn't worth the codegen complexity at this scale.
+ final StringBuilder lookupBody = new StringBuilder();
+ lookupBody.append("public ").append(holderFqcn)
+ .append(" debugHolder(String metricName) {\n");
+ for (CodeGenModel metric : dispatcherContext.getMetrics()) {
+ lookupBody.append(" if (\"")
+ .append(escapeJavaLiteral(metric.getTableName()))
+ .append("\".equals(metricName)) return this.debug_")
+ .append(metric.getTableName()).append(";\n");
+ }
+ lookupBody.append(" return null;\n}\n");
+ dispatcherClass.addMethod(CtNewMethod.make(lookupBody.toString(), dispatcherClass));
+
+ final StringBuilder namesBody = new StringBuilder();
+ namesBody.append("public String[] debugRuleNames() {\n")
+ .append(" return new String[] {");
+ boolean first = true;
+ for (CodeGenModel metric : dispatcherContext.getMetrics()) {
+ if (!first) {
+ namesBody.append(", ");
+ }
+ // Surface the snake_case OAL rule name — that's what operators
+ // see in their .oal files and what they pass on the install API.
+ namesBody.append("\"").append(escapeJavaLiteral(metric.getTableName())).append("\"");
+ first = false;
+ }
+ namesBody.append("};\n}\n");
+ dispatcherClass.addMethod(CtNewMethod.make(namesBody.toString(), dispatcherClass));
+ } catch (CannotCompileException e) {
+ log.error("Can't add DebugHolderProvider members on " + className + ".", e);
+ throw new OALCompileException(e.getMessage(), e);
+ } catch (NotFoundException nfe) {
+ log.error("DebugHolderProvider interface missing for " + className + ".", nfe);
+ throw new OALCompileException(nfe.getMessage(), nfe);
+ }
+ }
+
+ // Generate do<Metric>() per metric. Two FTL paths:
+ // - dispatcher/doMetrics.ftl — pre-SWIP-13 logic, byte-identical
+ // to a build without dsl-debugging
+ // - dispatcher/doMetricsWithDebug.ftl — same logic + per-stage probe call
+ // sites guarded on this.debug.isGateOn()
+ // Each FTL takes the CodeGenModel as-is — no extra context fields, no model
+ // wrapper. The codegen picks one or the other based on whether injection is
+ // enabled at boot; only one template runs per dispatcher.
+ final String doMetricsTemplate = DSLDebugCodegenSwitch.isInjectionEnabled()
+ ? "dispatcher/doMetricsWithDebug.ftl"
+ : "dispatcher/doMetrics.ftl";
+ final StringBuilder dispatcherSourceMethods = new StringBuilder();
for (CodeGenModel metric : dispatcherContext.getMetrics()) {
StringWriter methodEntity = new StringWriter();
try {
- configuration.getTemplate("dispatcher/doMetrics.ftl").process(metric, methodEntity);
- javassist.CtMethod m = CtNewMethod.make(methodEntity.toString(), dispatcherClass);
+ configuration.getTemplate(doMetricsTemplate).process(metric, methodEntity);
+ final String body = methodEntity.toString();
+ javassist.CtMethod m = CtNewMethod.make(body, dispatcherClass);
dispatcherClass.addMethod(m);
addLineNumberTable(m, 1);
+ dispatcherSourceMethods.append(" ").append(body.replace("\n", "\n ")).append("\n\n");
} catch (Exception e) {
log.error("Can't generate method do" + metric.getMetricsName() + " for " + className + ".", e);
log.error("Method body: {}", methodEntity);
@@ -399,9 +521,11 @@ public class OALClassGeneratorV2 {
try {
StringWriter methodEntity = new StringWriter();
configuration.getTemplate("dispatcher/dispatch.ftl").process(dispatcherContext, methodEntity);
- javassist.CtMethod m = CtNewMethod.make(methodEntity.toString(), dispatcherClass);
+ final String body = methodEntity.toString();
+ javassist.CtMethod m = CtNewMethod.make(body, dispatcherClass);
dispatcherClass.addMethod(m);
addLineNumberTable(m, 1);
+ dispatcherSourceMethods.append(" ").append(body.replace("\n", "\n ")).append("\n\n");
} catch (Exception e) {
log.error("Can't generate method dispatch for " + className + ".", e);
throw new OALCompileException(e.getMessage(), e);
@@ -429,9 +553,45 @@ public class OALClassGeneratorV2 {
}
writeGeneratedFile(dispatcherClass, "dispatcher");
+ writeGeneratedSourceFile(dispatcherClass, "dispatcher",
+ wrapDispatcherClassSource(dispatcherClass, dispatcherContext, dispatcherSourceMethods.toString()));
return targetClass;
}
+ /**
+ * Wraps the dispatcher's generated method bodies + GateHolder fields
+ * as a compilable class envelope. Renders the {@code debug_<metric>}
+ * fields when injection is enabled so the source mirrors the bytecode.
+ */
+ private String wrapDispatcherClassSource(final CtClass ctClass,
+ final DispatcherContextV2 ctx,
+ final String methods) {
+ final StringBuilder sb = new StringBuilder();
+ sb.append("package ").append(ctClass.getPackageName()).append(";\n\n");
+ sb.append("public class ").append(ctClass.getSimpleName())
+ .append(" implements org.apache.skywalking.oap.server.core.analysis.SourceDispatcher<")
+ .append(ctx.getSourcePackage()).append(ctx.getSourceName()).append("> {\n\n");
+ if (DSLDebugCodegenSwitch.isInjectionEnabled()) {
+ // The .java sidecar must compile cleanly for IDE source-attach. Final
+ // fields without initializers and stub method bodies would both
+ // reject — emit `null` initializers + return-null bodies. The actual
+ // bytecode this stands in for is produced by Javassist with full
+ // initializers in <clinit> and full method bodies; the sidecar only
+ // needs to round-trip through javac so source viewers can read it.
+ for (CodeGenModel metric : ctx.getMetrics()) {
+ sb.append(" public final ").append(GATE_HOLDER_CLASS).append(' ')
+ .append("debug_").append(metric.getTableName()).append(" = null;\n");
+ }
+ sb.append(" public ").append(GATE_HOLDER_CLASS)
+ .append(" debugHolder(String ruleName) { return null; }\n");
+ sb.append(" public String[] debugRuleNames() { return new String[0]; }\n");
+ sb.append('\n');
+ }
+ sb.append(methods);
+ sb.append("}\n");
+ return sb.toString();
+ }
+
private String metricsClassName(CodeGenModel model, boolean fullName) {
return (fullName ? oalDefine.getDynamicMetricsClassPackage() : "") + model.getMetricsName() + "Metrics";
}
@@ -498,6 +658,22 @@ public class OALClassGeneratorV2 {
* Adds a {@code LineNumberTable} attribute by scanning bytecode for
* store instructions to local variable slots ≥ {@code firstResultSlot}.
*/
+ /**
+ * Escape a string for embedding inside a Java source-string literal — same shape as
+ * the helpers in MAL / LAL codegen but kept local so this generator stays
+ * self-contained.
+ */
+ private static String escapeJavaLiteral(final String s) {
+ if (s == null) {
+ return "";
+ }
+ return s.replace("\\", "\\\\")
+ .replace("\"", "\\\"")
+ .replace("\n", "\\n")
+ .replace("\r", "\\r")
+ .replace("\t", "\\t");
+ }
+
private void addLineNumberTable(final javassist.CtMethod method,
final int firstResultSlot) {
try {
@@ -507,7 +683,7 @@ public class OALClassGeneratorV2 {
return;
}
- final java.util.ArrayList<int[]> entries = new java.util.ArrayList<>();
+ final ArrayList<int[]> entries = new ArrayList<>();
int line = 1;
boolean nextIsNewLine = true;
@@ -588,6 +764,40 @@ public class OALClassGeneratorV2 {
}
}
+ /**
+ * Sibling of {@link #writeGeneratedFile} that writes the Javassist-input
+ * Java source as {@code <ClassName>.java} alongside the {@code .class}.
+ * IDE source-attach renders this directly so the user sees the EXACT
+ * code Javassist compiled rather than a FernFlower decompile (which
+ * frequently bails on Javassist's bytecode patterns and leaves
+ * "compiled code" stubs). Caller assembles the Java source in their
+ * own buffer during method-body generation.
+ */
+ private void writeGeneratedSourceFile(final CtClass ctClass, final String type, final String javaSource) {
+ if (!openEngineDebug || javaSource == null) {
+ return;
+ }
+ try {
+ final File folder = new File(getGeneratedFilePath() + File.separator + type);
+ if (!folder.exists()) {
+ folder.mkdirs();
+ }
+ final File file = new File(folder, ctClass.getSimpleName() + ".java");
+ try (java.io.FileWriter w = new java.io.FileWriter(file)) {
+ w.write("// Synthetic source — Javassist compile input for ");
+ w.write(ctClass.getSimpleName());
+ w.write("\n// Written when SW_DYNAMIC_CLASS_ENGINE_DEBUG is on; used by IDE\n");
+ w.write("// source-attach to render the bytecode without FernFlower.\n\n");
+ w.write(javaSource);
+ if (!javaSource.endsWith("\n")) {
+ w.write("\n");
+ }
+ }
+ } catch (IOException e) {
+ log.warn("Can't write source file for " + ctClass.getSimpleName() + ", ignore.", e);
+ }
+ }
+
public void setStorageBuilderFactory(StorageBuilderFactory storageBuilderFactory) {
this.storageBuilderFactory = storageBuilderFactory;
}
@@ -607,6 +817,7 @@ public class OALClassGeneratorV2 {
this.openEngineDebug = debug;
}
+
/**
* V2 dispatcher context for grouping metrics by source.
*/
@@ -615,7 +826,7 @@ public class OALClassGeneratorV2 {
private String sourceName;
private String packageName;
private String sourceDecorator;
- private List<CodeGenModel> metrics = new java.util.ArrayList<>();
+ private List<CodeGenModel> metrics = new ArrayList<>();
public String getSourcePackage() {
return sourcePackage;
diff --git a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/model/FilterExpression.java b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/model/FilterExpression.java
index cd2ab14a56..ed4d0f9693 100644
--- a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/model/FilterExpression.java
+++ b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/model/FilterExpression.java
@@ -40,12 +40,23 @@ public final class FilterExpression {
private final FilterOperator operator;
private final FilterValue value;
private final SourceLocation location;
+ /**
+ * Verbatim source text of the filter clause from the original {@code .oal}
+ * file (e.g. {@code "filter(detectPoint == DetectPoint.SERVER)"}). Pulled
+ * from ANTLR's input stream at parse time, so whitespace and identifier
+ * spelling match the source byte-for-byte. Used as the {@code sourceText}
+ * of dsl-debugging captures so the UI matches a captured filter record
+ * against the original source byte-for-byte. Empty for filters constructed
+ * programmatically (test helpers); the codegen path tolerates {@code ""}.
+ */
+ private final String sourceText;
private FilterExpression(Builder builder) {
this.fieldName = Objects.requireNonNull(builder.fieldName, "fieldName cannot be null");
this.operator = Objects.requireNonNull(builder.operator, "operator cannot be null");
this.value = Objects.requireNonNull(builder.value, "value cannot be null");
this.location = builder.location != null ? builder.location : SourceLocation.UNKNOWN;
+ this.sourceText = builder.sourceText == null ? "" : builder.sourceText;
}
public static Builder builder() {
@@ -109,6 +120,12 @@ public final class FilterExpression {
private FilterOperator operator;
private FilterValue value;
private SourceLocation location;
+ private String sourceText;
+
+ public Builder sourceText(String sourceText) {
+ this.sourceText = sourceText;
+ return this;
+ }
public Builder fieldName(String fieldName) {
this.fieldName = fieldName;
diff --git a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/model/FunctionCall.java b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/model/FunctionCall.java
index 706cec5061..e40a84ad52 100644
--- a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/model/FunctionCall.java
+++ b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/model/FunctionCall.java
@@ -39,10 +39,22 @@ import lombok.Getter;
public final class FunctionCall {
private final String name;
private final List<FunctionArgument> arguments;
+ /**
+ * Verbatim source text of the function call from the original {@code .oal}
+ * file (e.g. {@code "cpm()"} / {@code "percentile2(10)"}). Pulled from
+ * ANTLR's input stream at parse time. Used as the {@code sourceText} of
+ * dsl-debugging captures for the aggregation stage.
+ */
+ private final String sourceText;
private FunctionCall(String name, List<FunctionArgument> arguments) {
+ this(name, arguments, "");
+ }
+
+ private FunctionCall(String name, List<FunctionArgument> arguments, String sourceText) {
this.name = Objects.requireNonNull(name, "function name cannot be null");
this.arguments = Collections.unmodifiableList(new ArrayList<>(arguments));
+ this.sourceText = sourceText == null ? "" : sourceText;
}
/**
@@ -92,12 +104,18 @@ public final class FunctionCall {
public static class Builder {
private String name;
private List<FunctionArgument> arguments = new ArrayList<>();
+ private String sourceText;
public Builder name(String name) {
this.name = name;
return this;
}
+ public Builder sourceText(String sourceText) {
+ this.sourceText = sourceText;
+ return this;
+ }
+
public Builder addArgument(FunctionArgument argument) {
this.arguments.add(argument);
return this;
@@ -124,7 +142,7 @@ public final class FunctionCall {
}
public FunctionCall build() {
- return new FunctionCall(name, arguments);
+ return new FunctionCall(name, arguments, sourceText);
}
}
}
diff --git a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/model/MetricDefinition.java b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/model/MetricDefinition.java
index 3050ee5dc0..9b5e68996c 100644
--- a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/model/MetricDefinition.java
+++ b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/model/MetricDefinition.java
@@ -44,6 +44,21 @@ public final class MetricDefinition {
private final FunctionCall aggregationFunction;
private final Optional<String> decorator;
private final SourceLocation location;
+ /**
+ * Verbatim text of this metric's full statement in the {@code .oal} file —
+ * the ANTLR Interval slice covering {@code <name> = from(...).filter(...).<func>();}.
+ * Threaded into the per-metric {@code GateHolder} so dsl-debugging records
+ * carry the rule source inline. {@code ""} for synthesised metrics.
+ */
+ private final String sourceText;
+ /**
+ * Verbatim text of just the {@code from(SourceName.attr)} /
+ * {@code from(SourceName.*)} segment — the ANTLR Interval slice from the
+ * FROM token through the closing source-paren. The dsl-debugging
+ * captureSource probe stamps this as the source sample's
+ * {@code sourceText} so the operator sees what they literally wrote.
+ */
+ private final String fromText;
private MetricDefinition(Builder builder) {
this.name = Objects.requireNonNull(builder.name, "name cannot be null");
@@ -53,6 +68,8 @@ public final class MetricDefinition {
this.aggregationFunction = Objects.requireNonNull(builder.aggregationFunction, "aggregationFunction cannot be null");
this.decorator = Optional.ofNullable(builder.decorator);
this.location = builder.location != null ? builder.location : SourceLocation.UNKNOWN;
+ this.sourceText = builder.sourceText == null ? "" : builder.sourceText;
+ this.fromText = builder.fromText == null ? "" : builder.fromText;
}
public static Builder builder() {
@@ -92,6 +109,18 @@ public final class MetricDefinition {
private FunctionCall aggregationFunction;
private String decorator;
private SourceLocation location;
+ private String sourceText;
+ private String fromText;
+
+ public Builder sourceText(String sourceText) {
+ this.sourceText = sourceText;
+ return this;
+ }
+
+ public Builder fromText(String fromText) {
+ this.fromText = fromText;
+ return this;
+ }
public Builder name(String name) {
this.name = name;
diff --git a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/parser/OALListenerV2.java b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/parser/OALListenerV2.java
index 11eb211b05..132ca8c766 100644
--- a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/parser/OALListenerV2.java
+++ b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/v2/parser/OALListenerV2.java
@@ -21,6 +21,8 @@ package org.apache.skywalking.oal.v2.parser;
import java.util.ArrayList;
import java.util.List;
import lombok.Getter;
+import org.antlr.v4.runtime.ParserRuleContext;
+import org.antlr.v4.runtime.misc.Interval;
import org.apache.skywalking.oal.rt.grammar.OALParser;
import org.apache.skywalking.oal.rt.grammar.OALParserBaseListener;
import org.apache.skywalking.oal.v2.model.FilterExpression;
@@ -99,10 +101,33 @@ public class OALListenerV2 extends OALParserBaseListener {
currentMetric.location(SourceLocation.of(fileName, line, column));
}
+ @Override
+ public void exitMetricStatement(OALParser.MetricStatementContext ctx) {
+ if (currentMetric != null) {
+ // Verbatim "from(SourceName.attr)" / "from(SourceName.*)" slice —
+ // ANTLR Interval from the FROM token through the FIRST closing
+ // paren that ends the source-attrs list. The dsl-debugging
+ // captureSource probe stamps this directly so the source sample's
+ // sourceText matches what the operator wrote in .oal byte-for-byte
+ // (no runtime reconstruction).
+ if (ctx.FROM() != null && ctx.RR_BRACKET() != null) {
+ final int start = ctx.FROM().getSymbol().getStartIndex();
+ final int stop = ctx.RR_BRACKET().getSymbol().getStopIndex();
+ currentMetric.fromText(
+ ctx.getStart().getInputStream().getText(Interval.of(start, stop)));
+ }
+ }
+ }
+
@Override
public void exitAggregationStatement(OALParser.AggregationStatementContext ctx) {
if (currentMetric != null && currentSource != null) {
currentMetric.source(currentSource.build());
+ // Verbatim ANTLR slice of the full assignment statement
+ // (e.g. "service_relation_server_cpm = from(ServiceRelation.*).filter(detectPoint == DetectPoint.SERVER).cpm();")
+ // — captured straight from the input stream, threaded into the
+ // per-metric GateHolder for dsl-debugging.
+ currentMetric.sourceText(rawTextOf(ctx));
metrics.add(currentMetric.build());
currentMetric = null;
currentSource = null;
@@ -188,6 +213,11 @@ public class OALListenerV2 extends OALParserBaseListener {
@Override
public void exitAggregateFunction(OALParser.AggregateFunctionContext ctx) {
if (currentFunction != null && currentMetric != null) {
+ // Verbatim ANTLR slice — e.g. "cpm()", "percentile2(10)" — captured
+ // exactly as written (including any whitespace inside the parens),
+ // so the dsl-debugging UI can match an aggregation capture record
+ // back to the original .oal byte-for-byte.
+ currentFunction.sourceText(rawTextOf(ctx));
currentMetric.aggregationFunction(currentFunction.build());
currentFunction = null;
}
@@ -531,11 +561,24 @@ public class OALListenerV2 extends OALParserBaseListener {
@Override
public void exitFilterStatement(OALParser.FilterStatementContext ctx) {
if (currentFilter != null && currentMetric != null) {
+ // Verbatim ANTLR slice — e.g. "filter(detectPoint == DetectPoint.SERVER)" —
+ // captured byte-for-byte from the input stream so the dsl-debugging UI
+ // can highlight the exact filter clause that fired in the original .oal.
+ currentFilter.sourceText(rawTextOf(ctx));
currentMetric.addFilter(currentFilter.build());
currentFilter = null;
}
}
+ /** Verbatim source slice of an ANTLR rule context. */
+ private static String rawTextOf(final ParserRuleContext ctx) {
+ if (ctx.getStart() == null || ctx.getStop() == null) {
+ return "";
+ }
+ return ctx.getStart().getInputStream().getText(
+ Interval.of(ctx.getStart().getStartIndex(), ctx.getStop().getStopIndex()));
+ }
+
@Override
public void enterDisableSource(OALParser.DisableSourceContext ctx) {
if (ctx.IDENTIFIER() != null) {
diff --git a/oap-server/oal-rt/src/main/resources/code-templates-v2/dispatcher/doMetricsWithDebug.ftl b/oap-server/oal-rt/src/main/resources/code-templates-v2/dispatcher/doMetricsWithDebug.ftl
new file mode 100644
index 0000000000..ab5953a111
--- /dev/null
+++ b/oap-server/oal-rt/src/main/resources/code-templates-v2/dispatcher/doMetricsWithDebug.ftl
@@ -0,0 +1,83 @@
+<#--
+ do<Metric>() with DSL-debug probe call sites. Identical pipeline shape to
+ doMetrics.ftl, plus gate-guarded captureXxx calls into OALDebug at every
+ stage. Selected by OALClassGeneratorV2 when DSLDebugCodegenSwitch reports
+ injection is enabled at boot. When off, doMetrics.ftl runs instead and the
+ generated bytecode is byte-identical to a build without SWIP-13.
+
+ Gate is per-metric: each do<Metric>() reads its own `this.debug_<metricsName>`
+ field. A session bound to one metric only fires that metric's probes — the
+ other metrics on the same dispatcher stay silent.
+-->
+private void do${metricsName}(${sourcePackage}${from.sourceName} source) {
+
+if (this.debug_${tableName}.isGateOn()) {
+ // sourceText is the verbatim "from(SourceName.attr)" / "from(SourceName.*)"
+ // slice — what the operator wrote in .oal — so the UI's source sample
+ // matches the original DSL byte-for-byte.
+ org.apache.skywalking.oap.server.core.dsldebug.OALDebug.captureSource(
+ this.debug_${tableName}, "${from.fromText?j_string}", ${sourceLine}, source);
+}
+
+<#if filters.filterExpressions??>
+ <#list filters.filterExpressions as filterExpression>
+ {
+ boolean _filterMatch_${filterExpression_index} =
+ new ${filterExpression.expressionObject}().match(${filterExpression.left}, ${filterExpression.right});
+ if (this.debug_${tableName}.isGateOn()) {
+ // Fire on every filter check, with continueOn = match result. Operators
+ // see both "this filter let this source through" (continueOn=true) and
+ // "this filter dropped it" (continueOn=false). sourceText is the verbatim
+ // ANTLR Interval slice (e.g. ".filter(detectPoint == DetectPoint.SERVER)").
+ org.apache.skywalking.oap.server.core.dsldebug.OALDebug.captureFilter(
+ this.debug_${tableName}, "${filterExpression.sourceText?j_string}", ${sourceLine},
+ source, _filterMatch_${filterExpression_index});
+ }
+ if (!_filterMatch_${filterExpression_index}) {
+ return;
+ }
+ }
+ </#list>
+</#if>
+
+${metricsClassPackage}${metricsName}Metrics metrics = new ${metricsClassPackage}${metricsName}Metrics();
+<#if sourceDecorator??>
+ source.decorate("${sourceDecorator}");
+</#if>
+metrics.setTimeBucket(source.getTimeBucket());
+<#list fieldsFromSource as field>
+ <#if field.attribute && !sourceDecorator??>
+ <#--Metrics share the source instance, do not process attributes unless added decorator func-->
+ <#else>
+ metrics.${field.fieldSetter}(source.${field.fieldGetter}());
+ </#if>
+</#list>
+
+if (this.debug_${tableName}.isGateOn()) {
+ // build: metric arg is the OAL rule's user-facing name (snake_case),
+ // matching what operators read in their .oal files.
+ org.apache.skywalking.oap.server.core.dsldebug.OALDebug.captureBuild(
+ this.debug_${tableName}, "${tableName}", ${sourceLine}, metrics);
+}
+
+metrics.${entryMethod.methodName}(
+<#list entryMethod.argsExpressions as arg>
+ <#if entryMethod.argTypes[arg_index] < 3>
+ ${arg}
+ <#else>
+ new ${arg.expressionObject}().match(${arg.left}, ${arg.right})
+ </#if><#if arg_has_next>, </#if>
+</#list>);
+
+if (this.debug_${tableName}.isGateOn()) {
+ // aggregation: sourceText is the verbatim OAL aggregation clause (e.g. "cpm()" /
+ // "percentile2(10)"). emit: sourceText is the OAL rule's user-facing name
+ // (snake_case) so the UI groups records by what operators wrote in their .oal.
+ org.apache.skywalking.oap.server.core.dsldebug.OALDebug.captureAggregation(
+ this.debug_${tableName}, "${tableName}", "${aggregationSourceText?j_string}", ${sourceLine}, metrics);
+ org.apache.skywalking.oap.server.core.dsldebug.OALDebug.captureEmit(
+ this.debug_${tableName}, "${tableName}", ${sourceLine}, metrics);
+}
+
+org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor.getInstance().in(metrics);
+}
diff --git a/oap-server/oal-rt/src/test/java/org/apache/skywalking/oal/v2/generator/RuntimeOALGenerationTest.java b/oap-server/oal-rt/src/test/java/org/apache/skywalking/oal/v2/generator/RuntimeOALGenerationTest.java
index 1665c5550e..85377e8f92 100644
--- a/oap-server/oal-rt/src/test/java/org/apache/skywalking/oal/v2/generator/RuntimeOALGenerationTest.java
+++ b/oap-server/oal-rt/src/test/java/org/apache/skywalking/oal/v2/generator/RuntimeOALGenerationTest.java
@@ -280,7 +280,16 @@ public class RuntimeOALGenerationTest {
models.add(model);
}
- // Create generator with OALDefine (catalog determines dispatcher class name prefix)
+ // Create generator with OALDefine (catalog determines dispatcher class name prefix).
+ // Mirror the production toggle so a developer can rebuild with
+ // SW_DSL_DEBUGGING_INJECTION_ENABLED=true and inspect the
+ // debug-probe-injected dispatcher classes (per-metric GateHolder
+ // fields + probe call sites) in target/test-classes/dispatcher/.
+ if ("true".equalsIgnoreCase(System.getenv("SW_DSL_DEBUGGING_INJECTION_ENABLED"))) {
+ org.apache.skywalking.oap.server.core.dsldebug.DSLDebugCodegenSwitch.enableInjection();
+ } else {
+ org.apache.skywalking.oap.server.core.dsldebug.DSLDebugCodegenSwitch.resetInjection();
+ }
OALClassGeneratorV2 generator = new OALClassGeneratorV2(define, classPool);
generator.setOpenEngineDebug(true);
generator.setStorageBuilderFactory(new StorageBuilderFactory.Default());
diff --git a/oap-server/pom.xml b/oap-server/pom.xml
index 93cc75452b..bb88dde9f1 100755
--- a/oap-server/pom.xml
+++ b/oap-server/pom.xml
@@ -30,6 +30,7 @@
<modules>
<module>server-core</module>
<module>analyzer</module>
+ <module>server-admin</module>
<module>server-receiver-plugin</module>
<module>server-cluster-plugin</module>
<module>server-storage-plugin</module>
diff --git a/oap-server/server-admin/admin-server/pom.xml b/oap-server/server-admin/admin-server/pom.xml
new file mode 100644
index 0000000000..84e8cfe865
--- /dev/null
+++ b/oap-server/server-admin/admin-server/pom.xml
@@ -0,0 +1,30 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-admin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>${revision}</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>admin-server</artifactId>
+ <packaging>jar</packaging>
+</project>
diff --git a/oap-server/server-admin/admin-server/src/main/java/org/apache/skywalking/oap/server/admin/server/cluster/AdminClusterChannelManager.java b/oap-server/server-admin/admin-server/src/main/java/org/apache/skywalking/oap/server/admin/server/cluster/AdminClusterChannelManager.java
new file mode 100644
index 0000000000..0e02464919
--- /dev/null
+++ b/oap-server/server-admin/admin-server/src/main/java/org/apache/skywalking/oap/server/admin/server/cluster/AdminClusterChannelManager.java
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.admin.server.cluster;
+
+import io.grpc.ManagedChannel;
+import java.util.List;
+import org.apache.skywalking.oap.server.library.module.Service;
+
+/**
+ * Maintains {@link ManagedChannel} connections to every reachable OAP peer at
+ * the admin-internal gRPC port. Module service exposed by {@code admin-server};
+ * admin features ({@code dsl-debugging}, {@code receiver-runtime-rule}) consume
+ * it for peer-to-peer cluster RPCs.
+ *
+ * <h2>Why a dedicated channel manager</h2>
+ * The cluster module's {@code RemoteClientManager} dials peers at the public
+ * agent / cluster gRPC port ({@code core.gRPCPort}, default 11800). Carrying
+ * privileged admin RPCs over that channel would put them in the same blast
+ * radius as agent telemetry: anyone with reach to the agent network could
+ * potentially invoke install / Suspend / Forward etc. Admin features use
+ * THIS manager instead — peers are dialed at the admin-internal port
+ * ({@code admin-server.gRPCPort}, default 17129), which operators bind to
+ * a private peer-to-peer interface.
+ *
+ * <h2>Peer discovery</h2>
+ * Hosts come from the cluster module's existing peer registry (zookeeper /
+ * k8s / nacos / standalone). Each peer's host is taken from the cluster
+ * registration; the port is the LOCAL admin-server config's
+ * {@code gRPCPort} — uniform across the cluster by convention. If a
+ * deployment uses non-uniform admin ports the cluster registration would
+ * need to publish the admin port per-peer, which is out of scope for the
+ * current phase.
+ *
+ * <h2>Channel lifecycle</h2>
+ * Channels are lazily built on first use per peer and cached. A cluster
+ * membership change (peer leaves) closes its channel; a new peer joining
+ * gets a fresh channel on next call. Self is filtered out by
+ * {@link Peer#isSelf()}.
+ */
+public interface AdminClusterChannelManager extends Service {
+
+ /**
+ * @return snapshot of every reachable peer's admin-side {@link Peer}.
+ * Self is included with {@link Peer#isSelf()} {@code true}; callers
+ * that fan out RPCs filter it out themselves.
+ */
+ List<Peer> getPeers();
+
+ /**
+ * Default per-call timeout (ms) for admin-internal RPCs, sourced from
+ * {@code admin-server.internalCommunicationTimeout}. Cluster clients that
+ * fan out install / collect / stop calls use this as the deadline; workflow-
+ * specific call sites (runtime-rule Suspend / Forward) keep their own
+ * tuned values.
+ */
+ long getInternalCommunicationTimeoutMs();
+
+ /**
+ * Per-peer view consumed by RPC fan-out helpers (dsl-debugging
+ * {@code ClusterPeerCaller}, runtime-rule {@code RuntimeRuleClusterClient}).
+ * The {@link #getChannel} return is {@code null} when the channel hasn't
+ * been established yet — callers report a transient {@code channel_not_ready}
+ * outcome and the next fan-out picks the channel up.
+ */
+ interface Peer {
+ /** Display address ({@code host:port}). */
+ String getAddress();
+
+ /** {@code true} when this entry is the local OAP node. */
+ boolean isSelf();
+
+ /** {@code null} until the channel is dialed. */
+ ManagedChannel getChannel();
+ }
+}
diff --git a/oap-server/server-admin/admin-server/src/main/java/org/apache/skywalking/oap/server/admin/server/cluster/AdminClusterChannelManagerImpl.java b/oap-server/server-admin/admin-server/src/main/java/org/apache/skywalking/oap/server/admin/server/cluster/AdminClusterChannelManagerImpl.java
new file mode 100644
index 0000000000..6da3791e42
--- /dev/null
+++ b/oap-server/server-admin/admin-server/src/main/java/org/apache/skywalking/oap/server/admin/server/cluster/AdminClusterChannelManagerImpl.java
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.admin.server.cluster;
+
+import io.grpc.ManagedChannel;
+import io.grpc.netty.GrpcSslContexts;
+import io.grpc.netty.NettyChannelBuilder;
+import io.netty.handler.ssl.SslContext;
+import java.io.File;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import javax.net.ssl.SSLException;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.cluster.ClusterNodesQuery;
+import org.apache.skywalking.oap.server.core.cluster.RemoteInstance;
+import org.apache.skywalking.oap.server.core.remote.client.Address;
+
+/**
+ * {@link AdminClusterChannelManager} backed by the cluster module's
+ * {@link ClusterNodesQuery} for peer discovery. Hosts come from the cluster
+ * registry; each peer is dialed at the LOCAL admin gRPC port (uniform
+ * across the cluster by convention). A background reconciler runs every
+ * {@code refreshIntervalMs} to open channels for new peers and shut down
+ * channels for departed peers.
+ *
+ * <p>Self detection: the cluster registry includes the local node and
+ * stamps {@link Address#isSelf()} on the local entry. We never open a
+ * channel to self — the {@link Peer#getChannel()} for the self entry
+ * returns {@code null}, and fan-out callers filter via
+ * {@link Peer#isSelf()} before calling.
+ *
+ * <p>Why a separate manager instead of riding {@code RemoteClientManager}:
+ * see {@link AdminClusterChannelManager} javadoc — {@code RemoteClientManager}
+ * dials peers at the public agent gRPC port; admin RPCs MUST stay off
+ * that channel.
+ */
+@Slf4j
+public class AdminClusterChannelManagerImpl implements AdminClusterChannelManager {
+
+ private static final long REFRESH_INTERVAL_MS = 5_000L;
+
+ /**
+ * Resolved lazily on first reconcile pass to support the framework's
+ * {@code requiredCheck} ordering. The check fires BEFORE every provider's
+ * {@code start()}, so this manager has to be REGISTERED from
+ * {@code prepare()} (else the admin-server provider fails its own
+ * required-services check). But {@link ClusterNodesQuery} comes from
+ * the cluster module, which isn't guaranteed to have its services
+ * available during admin-server's {@code prepare()}. The supplier
+ * defers the lookup until first reconcile, which fires from
+ * {@code notifyAfterCompleted()} after every module's {@code start()}.
+ */
+ private final Supplier<ClusterNodesQuery> clusterNodesQuerySupplier;
+ private volatile ClusterNodesQuery clusterNodesQuery;
+ private final int adminGrpcPort;
+ private final long internalCommunicationTimeoutMs;
+ private final SslContext clientSslContext;
+
+ /** Indexed by {@code host:adminPort}; access guarded by {@code synchronized(this)}. */
+ private final Map<String, ManagedChannel> channels = new HashMap<>();
+ private volatile List<Peer> snapshot = Collections.emptyList();
+
+ private final ScheduledExecutorService reconciler =
+ Executors.newSingleThreadScheduledExecutor(
+ r -> {
+ final Thread t = new Thread(r, "admin-peer-channel-reconciler");
+ t.setDaemon(true);
+ return t;
+ });
+
+ public AdminClusterChannelManagerImpl(final Supplier<ClusterNodesQuery> clusterNodesQuerySupplier,
+ final int adminGrpcPort,
+ final long internalCommunicationTimeoutMs,
+ final SslContext clientSslContext) {
+ this.clusterNodesQuerySupplier = clusterNodesQuerySupplier;
+ this.adminGrpcPort = adminGrpcPort;
+ this.internalCommunicationTimeoutMs = internalCommunicationTimeoutMs;
+ this.clientSslContext = clientSslContext;
+ }
+
+ @Override
+ public long getInternalCommunicationTimeoutMs() {
+ return internalCommunicationTimeoutMs;
+ }
+
+ /** Build the TLS context callers need to pass when {@code admin-server.gRPCSslEnabled=true}. */
+ public static SslContext clientSslContext(final String trustedCAsPath) throws SSLException {
+ return GrpcSslContexts.forClient()
+ .trustManager(new File(trustedCAsPath))
+ .build();
+ }
+
+ /**
+ * Run the first reconcile and schedule subsequent passes on the
+ * background thread. Called by {@code AdminServerModuleProvider}'s
+ * {@code notifyAfterCompleted} — never directly from feature modules.
+ */
+ public void start() {
+ reconcile();
+ reconciler.scheduleWithFixedDelay(this::reconcileQuiet,
+ REFRESH_INTERVAL_MS, REFRESH_INTERVAL_MS, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Stop the reconciler and close every open peer channel. Called on
+ * OAP shutdown via the module's lifecycle (or by tests that reuse the
+ * manager across phases).
+ */
+ public void shutdown() {
+ reconciler.shutdownNow();
+ synchronized (this) {
+ for (final ManagedChannel c : channels.values()) {
+ shutdownChannel(c);
+ }
+ channels.clear();
+ snapshot = Collections.emptyList();
+ }
+ }
+
+ @Override
+ public List<Peer> getPeers() {
+ return snapshot;
+ }
+
+ private void reconcileQuiet() {
+ try {
+ reconcile();
+ } catch (final Throwable t) {
+ log.warn("admin peer channel reconcile failed", t);
+ }
+ }
+
+ /**
+ * One reconcile pass:
+ * <ol>
+ * <li>Pull the current peer list from {@link ClusterNodesQuery}</li>
+ * <li>For new peers, dial a {@link ManagedChannel} at the admin port</li>
+ * <li>For departed peers, shut down their channel</li>
+ * <li>Publish a fresh immutable snapshot</li>
+ * </ol>
+ */
+ private synchronized void reconcile() {
+ if (clusterNodesQuery == null) {
+ clusterNodesQuery = clusterNodesQuerySupplier.get();
+ }
+ final List<RemoteInstance> nodes = clusterNodesQuery.queryRemoteNodes();
+ final Set<String> live = new HashSet<>();
+ final List<Peer> next = new ArrayList<>(nodes.size());
+
+ for (final RemoteInstance node : nodes) {
+ final String host = node.getAddress().getHost();
+ final boolean isSelf = node.getAddress().isSelf();
+ final String key = host + ":" + adminGrpcPort;
+ live.add(key);
+
+ ManagedChannel channel = null;
+ if (!isSelf) {
+ channel = channels.get(key);
+ if (channel == null) {
+ channel = buildChannel(host, adminGrpcPort);
+ if (channel != null) {
+ channels.put(key, channel);
+ log.info("admin peer channel opened to {}", key);
+ }
+ }
+ }
+ next.add(new ImmutablePeer(key, isSelf, channel));
+ }
+
+ // Close channels for peers no longer in the cluster.
+ final List<String> stale = new ArrayList<>();
+ for (final String key : channels.keySet()) {
+ if (!live.contains(key)) {
+ stale.add(key);
+ }
+ }
+ for (final String key : stale) {
+ shutdownChannel(channels.remove(key));
+ log.info("admin peer channel closed to {}", key);
+ }
+
+ snapshot = Collections.unmodifiableList(next);
+ }
+
+ private ManagedChannel buildChannel(final String host, final int port) {
+ try {
+ final NettyChannelBuilder builder = NettyChannelBuilder.forAddress(host, port);
+ if (clientSslContext != null) {
+ builder.useTransportSecurity().sslContext(clientSslContext);
+ } else {
+ builder.usePlaintext();
+ }
+ return builder.build();
+ } catch (final Throwable t) {
+ log.warn("failed to build admin peer channel to {}:{} — will retry next reconcile",
+ host, port, t);
+ return null;
+ }
+ }
+
+ private static void shutdownChannel(final ManagedChannel channel) {
+ if (channel == null) {
+ return;
+ }
+ try {
+ channel.shutdownNow().awaitTermination(2, TimeUnit.SECONDS);
+ } catch (final InterruptedException e) {
+ Thread.currentThread().interrupt();
+ } catch (final Throwable t) {
+ log.warn("admin peer channel shutdown raised", t);
+ }
+ }
+
+ private static final class ImmutablePeer implements Peer {
+ private final String address;
+ private final boolean isSelf;
+ private final ManagedChannel channel;
+
+ ImmutablePeer(final String address, final boolean isSelf,
+ final ManagedChannel channel) {
+ this.address = address;
+ this.isSelf = isSelf;
+ this.channel = channel;
+ }
+
+ @Override
+ public String getAddress() {
+ return address;
+ }
+
+ @Override
+ public boolean isSelf() {
+ return isSelf;
+ }
+
+ @Override
+ public ManagedChannel getChannel() {
+ return channel;
+ }
+ }
+}
diff --git a/oap-server/server-admin/admin-server/src/main/java/org/apache/skywalking/oap/server/admin/server/module/AdminServerModule.java b/oap-server/server-admin/admin-server/src/main/java/org/apache/skywalking/oap/server/admin/server/module/AdminServerModule.java
new file mode 100644
index 0000000000..e0db7d8f0a
--- /dev/null
+++ b/oap-server/server-admin/admin-server/src/main/java/org/apache/skywalking/oap/server/admin/server/module/AdminServerModule.java
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.admin.server.module;
+
+import org.apache.skywalking.oap.server.admin.server.cluster.AdminClusterChannelManager;
+import org.apache.skywalking.oap.server.core.server.GRPCHandlerRegister;
+import org.apache.skywalking.oap.server.core.server.HTTPHandlerRegister;
+import org.apache.skywalking.oap.server.library.module.ModuleDefine;
+
+/**
+ * Shared HTTP host for admin / on-demand write APIs (runtime-rule hot-update,
+ * DSL debug session, OAL read-only listing). Disabled by default; an operator
+ * opts in by setting {@code SW_ADMIN_SERVER=default}. Feature modules
+ * (runtime-rule, dsl-debugging) declare {@code AdminServerModule.NAME} in their
+ * {@code requiredModules()} and register their REST handlers via the
+ * {@link HTTPHandlerRegister} service this module exposes.
+ *
+ * <p>The admin port has NO built-in authentication. Operators MUST gateway-protect
+ * it with an IP allow-list and an authenticating reverse proxy, bind to a private
+ * interface, and never expose it to the public internet. See
+ * {@code docs/en/setup/backend/admin-api/readme.md} for the full security
+ * notice.
+ */
+public class AdminServerModule extends ModuleDefine {
+ public static final String NAME = "admin-server";
+
+ public AdminServerModule() {
+ super(NAME);
+ }
+
+ @Override
+ public Class[] services() {
+ return new Class[] {
+ HTTPHandlerRegister.class,
+ // Admin-internal gRPC infrastructure (default port 17129) — separate
+ // from the public agent / cluster gRPC port (default 11800) so
+ // privileged admin RPCs (dsl-debugging install/collect, runtime-rule
+ // Suspend/Resume/Forward) never share a blast radius with agent
+ // telemetry.
+ GRPCHandlerRegister.class,
+ AdminClusterChannelManager.class,
+ };
+ }
+}
diff --git a/oap-server/server-admin/admin-server/src/main/java/org/apache/skywalking/oap/server/admin/server/module/AdminServerModuleConfig.java b/oap-server/server-admin/admin-server/src/main/java/org/apache/skywalking/oap/server/admin/server/module/AdminServerModuleConfig.java
new file mode 100644
index 0000000000..3ebed11ffc
--- /dev/null
+++ b/oap-server/server-admin/admin-server/src/main/java/org/apache/skywalking/oap/server/admin/server/module/AdminServerModuleConfig.java
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.admin.server.module;
+
+import lombok.Getter;
+import lombok.Setter;
+import org.apache.skywalking.oap.server.library.module.ModuleConfig;
+
+@Getter
+@Setter
+public class AdminServerModuleConfig extends ModuleConfig {
+ /**
+ * Bind address for the admin HTTP host. Default {@code 0.0.0.0}.
+ * <p>SECURITY: set this to a private interface (or {@code 127.0.0.1}) — the
+ * admin port has no built-in authentication, so operators MUST gateway-protect
+ * the endpoints with an IP allow-list and an authenticating reverse proxy, and
+ * MUST NOT expose this address to the public internet.
+ */
+ private String host = "0.0.0.0";
+ /** Default {@code 17128}. Admin HTTP host port shared across feature modules. */
+ private int port = 17128;
+ private String contextPath = "/";
+ private int idleTimeOut = 30_000;
+ private int acceptQueueSize = 0;
+ private int httpMaxRequestHeaderSize = 8192;
+
+ /**
+ * Bind address for the admin-internal gRPC host that carries peer-to-peer
+ * cluster RPCs for admin features (dsl-debugging install/collect/stop,
+ * runtime-rule Suspend/Resume/Forward). Default {@code 0.0.0.0}.
+ * <p>SECURITY: this port MUST be reachable peer-to-peer between OAP nodes
+ * and MUST NEVER be exposed to the agent network or operators. It is
+ * intentionally separate from the public agent / cluster gRPC port
+ * ({@code core.gRPCPort}, default 11800) so an attacker on the agent
+ * network cannot invoke privileged admin RPCs.
+ */
+ private String gRPCHost = "0.0.0.0";
+
+ /**
+ * Default {@code 17129}. Internal gRPC port for peer-to-peer admin
+ * cluster RPCs. Bound only when the admin-server module is enabled.
+ * Every OAP in the cluster is assumed to use the same value — peers
+ * are discovered by host via the cluster module and dialed at this
+ * port. If you change it, change it on every node.
+ */
+ private int gRPCPort = 17129;
+
+ private int gRPCMaxConcurrentCallsPerConnection = 0;
+ private int gRPCMaxMessageSize = 50 * 1024 * 1024; // 50 MiB
+ private int gRPCThreadPoolSize = 0;
+ private int gRPCThreadPoolQueueSize = 0;
+ private boolean gRPCSslEnabled = false;
+ private String gRPCSslKeyPath = "";
+ private String gRPCSslCertChainPath = "";
+ private String gRPCSslTrustedCAsPath = "";
+
+ /**
+ * Default per-call timeout (ms) for admin-internal RPCs between OAP nodes —
+ * dsl-debugging {@code install / collect / stop} fan-outs and any other
+ * admin-internal RPC that doesn't override its own deadline. Workflow-specific
+ * call sites (runtime-rule {@code Suspend}, {@code Forward}) keep their own
+ * tuned values because their latency budgets are determined by the workflow,
+ * not by a generic timeout.
+ */
+ private int internalCommunicationTimeout = 5_000;
+}
diff --git a/oap-server/server-admin/admin-server/src/main/java/org/apache/skywalking/oap/server/admin/server/module/AdminServerModuleProvider.java b/oap-server/server-admin/admin-server/src/main/java/org/apache/skywalking/oap/server/admin/server/module/AdminServerModuleProvider.java
new file mode 100644
index 0000000000..4afed7db2a
--- /dev/null
+++ b/oap-server/server-admin/admin-server/src/main/java/org/apache/skywalking/oap/server/admin/server/module/AdminServerModuleProvider.java
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.admin.server.module;
+
+import io.netty.handler.ssl.SslContext;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.logging.log4j.util.Strings;
+import org.apache.skywalking.oap.server.admin.server.cluster.AdminClusterChannelManager;
+import org.apache.skywalking.oap.server.admin.server.cluster.AdminClusterChannelManagerImpl;
+import org.apache.skywalking.oap.server.core.CoreModule;
+import org.apache.skywalking.oap.server.core.RunningMode;
+import org.apache.skywalking.oap.server.core.cluster.ClusterModule;
+import org.apache.skywalking.oap.server.core.cluster.ClusterNodesQuery;
+import org.apache.skywalking.oap.server.core.server.GRPCHandlerRegister;
+import org.apache.skywalking.oap.server.core.server.GRPCHandlerRegisterImpl;
+import org.apache.skywalking.oap.server.core.server.HTTPHandlerRegister;
+import org.apache.skywalking.oap.server.core.server.HTTPHandlerRegisterImpl;
+import org.apache.skywalking.oap.server.library.module.ModuleDefine;
+import org.apache.skywalking.oap.server.library.module.ModuleProvider;
+import org.apache.skywalking.oap.server.library.module.ModuleStartException;
+import org.apache.skywalking.oap.server.library.server.ServerException;
+import org.apache.skywalking.oap.server.library.server.grpc.GRPCServer;
+import org.apache.skywalking.oap.server.library.server.http.HTTPServer;
+import org.apache.skywalking.oap.server.library.server.http.HTTPServerConfig;
+
+/**
+ * Owns the Armeria HTTP host shared by every admin / on-demand write feature.
+ * The provider is loaded only when an operator enables the module via the
+ * {@code SW_ADMIN_SERVER=default} selector. Until then the admin port never
+ * opens — feature modules that depend on {@link AdminServerModule#NAME} fail
+ * fast at boot with a {@code ModuleNotFoundException}.
+ *
+ * <p>The provider exposes {@link HTTPHandlerRegister} as a service. Feature
+ * modules ({@code receiver-runtime-rule}, {@code dsl-debugging}) call
+ * {@code moduleManager.find(AdminServerModule.NAME).provider().getService(HTTPHandlerRegister.class)}
+ * during their {@code start()} phase and add their own handlers. The HTTP
+ * server itself is initialized in {@link #prepare()} and started in
+ * {@link #notifyAfterCompleted()} — the same lifecycle the other Armeria-hosting
+ * providers in OAP follow, so handlers added during {@code start()} are picked
+ * up when the server actually accepts connections.
+ *
+ * <p>SECURITY: the admin port has no built-in authentication. Operators MUST
+ * gateway-protect the endpoints, bind to a private interface, and never expose
+ * the port to the public internet. See
+ * {@code docs/en/setup/backend/admin-api/readme.md}.
+ */
+@Slf4j
+public class AdminServerModuleProvider extends ModuleProvider {
+
+ private AdminServerModuleConfig moduleConfig;
+ private HTTPServer httpServer;
+ private GRPCServer grpcServer;
+ private AdminClusterChannelManagerImpl peerChannelManager;
+
+ @Override
+ public String name() {
+ return "default";
+ }
+
+ @Override
+ public Class<? extends ModuleDefine> module() {
+ return AdminServerModule.class;
+ }
+
+ @Override
+ public ConfigCreator newConfigCreator() {
+ return new ConfigCreator<AdminServerModuleConfig>() {
+ @Override
+ public Class type() {
+ return AdminServerModuleConfig.class;
+ }
+
+ @Override
+ public void onInitialized(final AdminServerModuleConfig initialized) {
+ moduleConfig = initialized;
+ }
+ };
+ }
+
+ @Override
+ public void prepare() {
+ if (moduleConfig.getPort() <= 0) {
+ throw new IllegalStateException(
+ "admin-server: port must be > 0 when the module is enabled, got "
+ + moduleConfig.getPort());
+ }
+ final HTTPServerConfig httpServerConfig =
+ HTTPServerConfig.builder()
+ .host(Strings.isBlank(moduleConfig.getHost()) ? "0.0.0.0"
+ : moduleConfig.getHost())
+ .port(moduleConfig.getPort())
+ .contextPath(moduleConfig.getContextPath())
+ .acceptQueueSize(moduleConfig.getAcceptQueueSize())
+ .idleTimeOut(moduleConfig.getIdleTimeOut())
+ .maxRequestHeaderSize(moduleConfig.getHttpMaxRequestHeaderSize())
+ .build();
+ httpServer = new HTTPServer(httpServerConfig);
+ httpServer.setBlockingTaskName("admin-http");
+ httpServer.initialize();
+ registerServiceImplementation(HTTPHandlerRegister.class,
+ new HTTPHandlerRegisterImpl(httpServer));
+
+ // Admin-internal gRPC server — peer-to-peer cluster RPCs for admin
+ // features. Bound separately from the public agent / cluster gRPC
+ // port (default 11800) so privileged admin RPCs (dsl-debugging
+ // install/collect, runtime-rule Suspend/Resume/Forward) never share
+ // a blast radius with agent telemetry. Operators bind this to a
+ // private peer-to-peer interface; the cluster module dials each
+ // peer at this port via AdminClusterChannelManager.
+ if (moduleConfig.getGRPCPort() <= 0) {
+ throw new IllegalStateException(
+ "admin-server: gRPCPort must be > 0 when the module is enabled, got "
+ + moduleConfig.getGRPCPort());
+ }
+ if (moduleConfig.isGRPCSslEnabled()) {
+ grpcServer = new GRPCServer(
+ Strings.isBlank(moduleConfig.getGRPCHost()) ? "0.0.0.0" : moduleConfig.getGRPCHost(),
+ moduleConfig.getGRPCPort(),
+ moduleConfig.getGRPCSslCertChainPath(),
+ moduleConfig.getGRPCSslKeyPath(),
+ moduleConfig.getGRPCSslTrustedCAsPath());
+ } else {
+ grpcServer = new GRPCServer(
+ Strings.isBlank(moduleConfig.getGRPCHost()) ? "0.0.0.0" : moduleConfig.getGRPCHost(),
+ moduleConfig.getGRPCPort());
+ }
+ if (moduleConfig.getGRPCMaxConcurrentCallsPerConnection() > 0) {
+ grpcServer.setMaxConcurrentCallsPerConnection(
+ moduleConfig.getGRPCMaxConcurrentCallsPerConnection());
+ }
+ if (moduleConfig.getGRPCMaxMessageSize() > 0) {
+ grpcServer.setMaxMessageSize(moduleConfig.getGRPCMaxMessageSize());
+ }
+ if (moduleConfig.getGRPCThreadPoolSize() > 0) {
+ grpcServer.setThreadPoolSize(moduleConfig.getGRPCThreadPoolSize());
+ }
+ grpcServer.setThreadPoolName("admin-grpc");
+ grpcServer.initialize();
+ registerServiceImplementation(GRPCHandlerRegister.class,
+ new GRPCHandlerRegisterImpl(grpcServer));
+
+ // Peer channel manager: register IMMEDIATELY in prepare() so the
+ // framework's requiredCheck (which fires before any provider's
+ // start()) sees this service. ClusterNodesQuery isn't available
+ // here yet — pass a supplier that resolves it lazily on first
+ // reconcile, which fires from notifyAfterCompleted() after every
+ // module's start() has run.
+ // When the server side is TLS-enabled, the client side MUST also use TLS — a
+ // missing or empty trusted-CAs path on the client would leave us dialling peers
+ // in plaintext at a port the server is only willing to handshake. Fail fast at
+ // boot rather than letting the cluster silently break at first reconcile.
+ SslContext clientSslContext = null;
+ if (moduleConfig.isGRPCSslEnabled()) {
+ if (moduleConfig.getGRPCSslTrustedCAsPath() == null
+ || moduleConfig.getGRPCSslTrustedCAsPath().isEmpty()) {
+ throw new IllegalStateException(
+ "admin-server: gRPCSslEnabled=true but gRPCSslTrustedCAsPath is empty. "
+ + "The admin-internal gRPC bus needs a CA bundle on every node so "
+ + "peer channels can establish TLS to the server's cert. Set "
+ + "SW_ADMIN_SERVER_GRPC_SSL_TRUSTED_CAS_PATH (or "
+ + "admin-server.gRPCSslTrustedCAsPath in application.yml) on every "
+ + "OAP, or set gRPCSslEnabled=false everywhere.");
+ }
+ try {
+ clientSslContext = AdminClusterChannelManagerImpl.clientSslContext(
+ moduleConfig.getGRPCSslTrustedCAsPath());
+ } catch (final Exception e) {
+ throw new IllegalStateException(
+ "admin-server: failed to build admin gRPC client SSL context", e);
+ }
+ }
+ peerChannelManager = new AdminClusterChannelManagerImpl(
+ () -> getManager().find(ClusterModule.NAME).provider()
+ .getService(ClusterNodesQuery.class),
+ moduleConfig.getGRPCPort(),
+ moduleConfig.getInternalCommunicationTimeout(),
+ clientSslContext);
+ registerServiceImplementation(AdminClusterChannelManager.class, peerChannelManager);
+ }
+
+ @Override
+ public void start() {
+ // Routes are added by feature modules in their start() phase via the
+ // HTTPHandlerRegister service exposed above. Channel manager is
+ // already registered in prepare() with a lazy ClusterNodesQuery
+ // supplier; nothing else to do here.
+ }
+
+ @Override
+ public void notifyAfterCompleted() throws ModuleStartException {
+ if (RunningMode.isInitMode()) {
+ return;
+ }
+ try {
+ if (grpcServer != null) {
+ grpcServer.start();
+ log.info("admin-server gRPC listening on {}:{} (peer-to-peer admin RPCs only — "
+ + "MUST be reachable between OAP nodes; MUST NOT be exposed to the agent network "
+ + "or operators).",
+ moduleConfig.getGRPCHost(), moduleConfig.getGRPCPort());
+ }
+ } catch (final ServerException e) {
+ throw new ModuleStartException("admin-server: failed to start gRPC server", e);
+ }
+ if (peerChannelManager != null) {
+ peerChannelManager.start();
+ }
+ if (httpServer != null) {
+ httpServer.start();
+ log.info(
+ "admin-server HTTP listening on {}:{} (no built-in authentication — "
+ + "gateway-protect with an IP allow-list and authenticating reverse proxy; "
+ + "never expose to the public internet).",
+ moduleConfig.getHost(), moduleConfig.getPort()
+ );
+ }
+ }
+
+ @Override
+ public String[] requiredModules() {
+ return new String[] {
+ CoreModule.NAME, // GRPCServer + GRPCHandlerRegisterImpl live here
+ ClusterModule.NAME, // ClusterNodesQuery for peer discovery
+ };
+ }
+}
diff --git a/oap-server/server-admin/admin-server/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleDefine b/oap-server/server-admin/admin-server/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleDefine
new file mode 100644
index 0000000000..3d9d0988af
--- /dev/null
+++ b/oap-server/server-admin/admin-server/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleDefine
@@ -0,0 +1 @@
+org.apache.skywalking.oap.server.admin.server.module.AdminServerModule
diff --git a/oap-server/server-admin/admin-server/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleProvider b/oap-server/server-admin/admin-server/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleProvider
new file mode 100644
index 0000000000..aa7663897e
--- /dev/null
+++ b/oap-server/server-admin/admin-server/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleProvider
@@ -0,0 +1 @@
+org.apache.skywalking.oap.server.admin.server.module.AdminServerModuleProvider
diff --git a/oap-server/server-receiver-plugin/skywalking-runtime-rule-receiver-plugin/pom.xml b/oap-server/server-admin/dsl-debugging/pom.xml
similarity index 51%
copy from oap-server/server-receiver-plugin/skywalking-runtime-rule-receiver-plugin/pom.xml
copy to oap-server/server-admin/dsl-debugging/pom.xml
index da9ae62e02..954d05ba1d 100644
--- a/oap-server/server-receiver-plugin/skywalking-runtime-rule-receiver-plugin/pom.xml
+++ b/oap-server/server-admin/dsl-debugging/pom.xml
@@ -19,55 +19,36 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
- <artifactId>server-receiver-plugin</artifactId>
+ <artifactId>server-admin</artifactId>
<groupId>org.apache.skywalking</groupId>
<version>${revision}</version>
</parent>
<modelVersion>4.0.0</modelVersion>
- <artifactId>skywalking-runtime-rule-receiver-plugin</artifactId>
+ <artifactId>dsl-debugging</artifactId>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.apache.skywalking</groupId>
- <artifactId>server-core</artifactId>
+ <artifactId>admin-server</artifactId>
<version>${project.version}</version>
</dependency>
- <dependency>
- <groupId>org.apache.skywalking</groupId>
- <artifactId>library-server</artifactId>
- <version>${project.version}</version>
- </dependency>
- <!--
- meter-analyzer carries MetricConvert / Rule / Analyzer — the MAL compile pipeline.
- The apply step constructs one MetricConvert per rule-file bundle, which invokes the
- generator + MeterSystem.create per declared metric name. Scoped compile+runtime.
- -->
<dependency>
<groupId>org.apache.skywalking</groupId>
<artifactId>meter-analyzer</artifactId>
<version>${project.version}</version>
</dependency>
- <!--
- log-analyzer carries LogFilterListener.Factory and the LAL compile helpers. The
- runtime-rule LAL apply path reuses Factory.compile + Factory.addOrReplace / remove
- so every hot-update hits the same registry the runtime log pipeline consults.
- -->
<dependency>
<groupId>org.apache.skywalking</groupId>
<artifactId>log-analyzer</artifactId>
<version>${project.version}</version>
</dependency>
- <!-- commons-compress provides TarArchiveOutputStream for /runtime/rule/dump. -->
- <dependency>
- <groupId>org.apache.commons</groupId>
- <artifactId>commons-compress</artifactId>
- </dependency>
<!--
- gRPC dependencies needed to generate and host RuntimeRuleClusterService. The Suspend
- RPC is a small cluster-internal call that travels between OAP nodes on the existing
- inter-node transport; compile-level deps only, no netty shading concerns here.
+ gRPC dependencies for the cluster fan-out RPCs (InstallDebugSession,
+ CollectDebugSamples, StopDebugSession, StopByClientId). Travel over
+ the admin-internal gRPC bus owned by admin-server (default port 17129),
+ NOT the public agent / cluster bus (default 11800).
-->
<dependency>
<groupId>io.grpc</groupId>
@@ -79,51 +60,15 @@
<artifactId>grpc-stub</artifactId>
<version>${grpc.version}</version>
</dependency>
- <!--
- IT-only deps. library-integration-test provides BanyanDBTestContainer + ITVersions
- so the runtime-rule ITs share image/port/version constants with every other BanyanDB
- IT in the repo. library-banyandb-client gives MalApplyRollbackIT / MalShapeBreakIT a
- read-side verifyClient to inspect groups and measures independently of the
- storage-plugin write path.
- -->
- <dependency>
- <groupId>org.apache.skywalking</groupId>
- <artifactId>library-integration-test</artifactId>
- <version>${project.version}</version>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.skywalking</groupId>
- <artifactId>library-banyandb-client</artifactId>
- <version>${project.version}</version>
- <scope>test</scope>
- </dependency>
- <!--
- storage-banyandb-plugin gives us BanyanDBIndexInstaller, BanyanDBStorageDAO,
- BanyanDBBatchDAO, and BanyanDBRuntimeRuleManagementDAO — the concrete services the
- StorageStackHarness wires into the mocked ModuleManager so MeterSystem.create fires
- real DDL against the Testcontainers BanyanDB.
- -->
- <dependency>
- <groupId>org.apache.skywalking</groupId>
- <artifactId>storage-banyandb-plugin</artifactId>
- <version>${project.version}</version>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.testcontainers</groupId>
- <artifactId>testcontainers</artifactId>
- <scope>test</scope>
- </dependency>
<dependency>
- <groupId>org.testcontainers</groupId>
+ <groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<scope>test</scope>
</dependency>
<!--
- grpc-testing provides InProcessServerBuilder / InProcessChannelBuilder used by the
- two-node cluster harness. Keeps the in-memory gRPC transport real (deadlines, status
- codes, cancellation) without a TCP loopback.
+ grpc-testing provides InProcessServerBuilder / InProcessChannelBuilder
+ for the cluster RPC integration tests — keeps the gRPC transport real
+ (deadlines, status codes) without TCP loopback.
-->
<dependency>
<groupId>io.grpc</groupId>
diff --git a/oap-server/server-admin/dsl-debugging/src/main/java/org/apache/skywalking/oap/server/admin/dsl/debugging/cluster/ClusterPeerCaller.java b/oap-server/server-admin/dsl-debugging/src/main/java/org/apache/skywalking/oap/server/admin/dsl/debugging/cluster/ClusterPeerCaller.java
new file mode 100644
index 0000000000..8b2571f165
--- /dev/null
+++ b/oap-server/server-admin/dsl-debugging/src/main/java/org/apache/skywalking/oap/server/admin/dsl/debugging/cluster/ClusterPeerCaller.java
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.admin.dsl.debugging.cluster;
+
+import io.grpc.ManagedChannel;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Function;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.admin.server.cluster.AdminClusterChannelManager;
+
+/**
+ * Per-peer fan-out helper. Same shape every cluster RPC needs — iterate
+ * {@link AdminClusterChannelManager#getPeers()}, skip self, invoke the
+ * per-peer body under the per-call deadline, wrap the result (or the
+ * failure reason) into a {@link PeerOutcome}. Returning a list of typed
+ * outcomes lets the receiving REST handler aggregate without scattered
+ * try/catch or null-check branches.
+ *
+ * <p>Channels come from the admin-internal gRPC bus (default port 17129),
+ * NOT the public agent / cluster bus (default 11800). See
+ * {@link AdminClusterChannelManager} for the rationale.
+ *
+ * <p>Fan-out runs in parallel across peers, with each per-peer call still
+ * carrying its own gRPC deadline. The wall-clock latency of the whole fan-out
+ * is therefore bounded by the slowest reachable peer's deadline, not the
+ * sum across peers — important for collect on a partitioned 10-node cluster
+ * where a sequential fan-out would multiply the deadline per missed peer.
+ */
+@Slf4j
+public final class ClusterPeerCaller {
+
+ private static final AtomicInteger THREAD_SEQ = new AtomicInteger();
+
+ private final AdminClusterChannelManager peerChannelManager;
+ private final String selfNodeId;
+
+ /**
+ * Bounded executor shared by every fan-out from this caller. Each task is
+ * one peer's blocking gRPC call, so threads are I/O-bound rather than
+ * CPU-bound; a fixed pool sized to the typical peer count is enough and
+ * caps the worst-case live-thread footprint when many fan-outs queue up.
+ */
+ private final ExecutorService executor = Executors.newFixedThreadPool(16, r -> {
+ final Thread t = new Thread(r, "admin-cluster-fanout-" + THREAD_SEQ.incrementAndGet());
+ t.setDaemon(true);
+ return t;
+ });
+
+ public ClusterPeerCaller(final AdminClusterChannelManager peerChannelManager,
+ final String selfNodeId) {
+ this.peerChannelManager = peerChannelManager;
+ this.selfNodeId = selfNodeId;
+ }
+
+ public String getSelfNodeId() {
+ return selfNodeId;
+ }
+
+ /**
+ * Fan out one RPC call to every non-self peer in parallel.
+ *
+ * @param description short description (logged on failure, included in failure detail)
+ * @param body per-peer body — receives the peer's {@link ManagedChannel} (never
+ * null; the helper skips peers without an established channel) and
+ * returns the typed ack payload. Throwing from this body is folded
+ * into a {@link PeerOutcome#failed(String, String)} entry.
+ */
+ public <A> List<PeerOutcome<A>> fanOut(final String description,
+ final Function<ManagedChannel, A> body) {
+ final List<AdminClusterChannelManager.Peer> peers = peerChannelManager.getPeers();
+ final List<CompletableFuture<PeerOutcome<A>>> futures = new ArrayList<>(peers.size());
+ for (final AdminClusterChannelManager.Peer peer : peers) {
+ if (peer.isSelf()) {
+ continue;
+ }
+ futures.add(CompletableFuture.supplyAsync(
+ () -> invokeOne(peer, description, body), executor));
+ }
+ final long globalDeadlineMs = peerChannelManager.getInternalCommunicationTimeoutMs();
+ final long deadlineNanos = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(globalDeadlineMs);
+ final List<PeerOutcome<A>> outcomes = new ArrayList<>(futures.size());
+ for (final CompletableFuture<PeerOutcome<A>> future : futures) {
+ final long remaining = Math.max(0L, deadlineNanos - System.nanoTime());
+ try {
+ outcomes.add(future.get(remaining, TimeUnit.NANOSECONDS));
+ } catch (final TimeoutException te) {
+ future.cancel(true);
+ outcomes.add(PeerOutcome.failed("unknown", "fanout_global_deadline_exceeded"));
+ } catch (final InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ future.cancel(true);
+ outcomes.add(PeerOutcome.failed("unknown", "interrupted"));
+ } catch (final ExecutionException ee) {
+ outcomes.add(PeerOutcome.failed("unknown", ee.getCause() == null
+ ? ee.getMessage() : ee.getCause().getMessage()));
+ }
+ }
+ return outcomes;
+ }
+
+ private <A> PeerOutcome<A> invokeOne(final AdminClusterChannelManager.Peer peer, final String description,
+ final Function<ManagedChannel, A> body) {
+ final String peerAddress = peer.getAddress();
+ final ManagedChannel channel = peer.getChannel();
+ if (channel == null) {
+ log.warn("DSL debug {} skipped for peer {}: channel not yet established",
+ description, peerAddress);
+ return PeerOutcome.failed(peerAddress, "channel_not_ready");
+ }
+ try {
+ final A ack = body.apply(channel);
+ if (ack == null) {
+ return PeerOutcome.failed(peerAddress, "null_response");
+ }
+ return PeerOutcome.ok(peerAddress, ack);
+ } catch (final Throwable t) {
+ log.warn("DSL debug {} to peer {} failed: {}", description, peerAddress, t.getMessage());
+ return PeerOutcome.failed(peerAddress, t.getMessage());
+ }
+ }
+}
diff --git a/oap-server/server-admin/dsl-debugging/src/main/java/org/apache/skywalking/oap/server/admin/dsl/debugging/cluster/DSLDebuggingClusterClient.java b/oap-server/server-admin/dsl-debugging/src/main/java/org/apache/skywalking/oap/server/admin/dsl/debugging/cluster/DSLDebuggingClusterClient.java
new file mode 100644
index 0000000000..6987d87634
--- /dev/null
+++ b/oap-server/server-admin/dsl-debugging/src/main/java/org/apache/skywalking/oap/server/admin/dsl/debugging/cluster/DSLDebuggingClusterClient.java
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.admin.dsl.debugging.cluster;
+
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import lombok.RequiredArgsConstructor;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.ClusterRuleKey;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.ClusterSessionLimits;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.CollectDebugSamplesAck;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.CollectDebugSamplesRequest;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.DSLDebuggingClusterServiceGrpc;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.InstallDebugSessionAck;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.InstallDebugSessionRequest;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.StopByClientIdAck;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.StopByClientIdRequest;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.StopDebugSessionAck;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.StopDebugSessionRequest;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.session.SessionLimits;
+import org.apache.skywalking.oap.server.core.dsldebug.RuleKey;
+
+/**
+ * Outbound cluster broadcasts for the four debug-session RPCs.
+ *
+ * <h2>Workflow</h2>
+ * <pre>
+ * POST /dsl-debugging/session
+ * │
+ * ├─ stopByClientId(clientId) broadcast cleanup of any prior session under
+ * │ this clientId on every peer (per SWIP §6).
+ * ├─ install(sessionId, ruleKey, ...) broadcast install to every peer; each peer
+ * │ binds a local recorder to its own holder.
+ * ▼
+ *
+ * GET /dsl-debugging/session/{id}
+ * └─ collect(sessionId) broadcast collect; receiving node concatenates
+ * per-node slices into the response nodes[] array.
+ *
+ * POST /dsl-debugging/session/{id}/stop
+ * └─ stop(sessionId) best-effort broadcast; missed peers fall out via
+ * retention timeout.
+ * </pre>
+ *
+ * <p>Each broadcast returns one {@link PeerOutcome} per non-self peer; the
+ * receiving node's REST handler aggregates them under the SWIP §1 response
+ * shape.
+ */
+@RequiredArgsConstructor
+public final class DSLDebuggingClusterClient {
+
+ private final ClusterPeerCaller peerCaller;
+ private final long perCallDeadlineMs;
+ private final long collectDeadlineMs;
+
+ public List<PeerOutcome<InstallDebugSessionAck>> broadcastInstall(
+ final String sessionId, final String clientId, final RuleKey ruleKey,
+ final SessionLimits limits) {
+ final InstallDebugSessionRequest req = InstallDebugSessionRequest.newBuilder()
+ .setSessionId(sessionId)
+ .setClientId(clientId == null ? "" : clientId)
+ .setRuleKey(toClusterRuleKey(ruleKey))
+ .setLimits(toClusterLimits(limits))
+ .setSenderNodeId(peerCaller.getSelfNodeId())
+ .setIssuedAtMs(System.currentTimeMillis())
+ .build();
+ return peerCaller.fanOut("InstallDebugSession", channel ->
+ DSLDebuggingClusterServiceGrpc.newBlockingStub(channel)
+ .withDeadlineAfter(perCallDeadlineMs, TimeUnit.MILLISECONDS)
+ .installDebugSession(req));
+ }
+
+ public List<PeerOutcome<CollectDebugSamplesAck>> broadcastCollect(final String sessionId) {
+ final CollectDebugSamplesRequest req = CollectDebugSamplesRequest.newBuilder()
+ .setSessionId(sessionId)
+ .setSenderNodeId(peerCaller.getSelfNodeId())
+ .build();
+ return peerCaller.fanOut("CollectDebugSamples", channel ->
+ DSLDebuggingClusterServiceGrpc.newBlockingStub(channel)
+ .withDeadlineAfter(collectDeadlineMs, TimeUnit.MILLISECONDS)
+ .collectDebugSamples(req));
+ }
+
+ public List<PeerOutcome<StopDebugSessionAck>> broadcastStop(final String sessionId) {
+ final StopDebugSessionRequest req = StopDebugSessionRequest.newBuilder()
+ .setSessionId(sessionId)
+ .setSenderNodeId(peerCaller.getSelfNodeId())
+ .build();
+ return peerCaller.fanOut("StopDebugSession", channel ->
+ DSLDebuggingClusterServiceGrpc.newBlockingStub(channel)
+ .withDeadlineAfter(perCallDeadlineMs, TimeUnit.MILLISECONDS)
+ .stopDebugSession(req));
+ }
+
+ public List<PeerOutcome<StopByClientIdAck>> broadcastStopByClientId(final String clientId) {
+ final StopByClientIdRequest req = StopByClientIdRequest.newBuilder()
+ .setClientId(clientId == null ? "" : clientId)
+ .setSenderNodeId(peerCaller.getSelfNodeId())
+ .build();
+ return peerCaller.fanOut("StopByClientId", channel ->
+ DSLDebuggingClusterServiceGrpc.newBlockingStub(channel)
+ .withDeadlineAfter(perCallDeadlineMs, TimeUnit.MILLISECONDS)
+ .stopByClientId(req));
+ }
+
+ private static ClusterRuleKey toClusterRuleKey(final RuleKey ruleKey) {
+ return ClusterRuleKey.newBuilder()
+ .setCatalog(ruleKey.getCatalog().getWireName())
+ .setName(ruleKey.getName())
+ .setRuleName(ruleKey.getRuleName())
+ .build();
+ }
+
+ private static ClusterSessionLimits toClusterLimits(final SessionLimits limits) {
+ return ClusterSessionLimits.newBuilder()
+ .setRecordCap(limits.getRecordCap())
+ .setRetentionMillis(limits.getRetentionMillis())
+ .setGranularity(limits.getGranularity().wireName())
+ .build();
+ }
+}
diff --git a/oap-server/server-admin/dsl-debugging/src/main/java/org/apache/skywalking/oap/server/admin/dsl/debugging/cluster/DSLDebuggingClusterServiceImpl.java b/oap-server/server-admin/dsl-debugging/src/main/java/org/apache/skywalking/oap/server/admin/dsl/debugging/cluster/DSLDebuggingClusterServiceImpl.java
new file mode 100644
index 0000000000..f1badcd6e8
--- /dev/null
+++ b/oap-server/server-admin/dsl-debugging/src/main/java/org/apache/skywalking/oap/server/admin/dsl/debugging/cluster/DSLDebuggingClusterServiceImpl.java
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.admin.dsl.debugging.cluster;
+
+import io.grpc.stub.StreamObserver;
+import java.util.List;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.ClusterExecutionRecord;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.ClusterRuleKey;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.ClusterSample;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.ClusterSessionLimits;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.CollectDebugSamplesAck;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.CollectDebugSamplesRequest;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.DSLDebuggingClusterServiceGrpc;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.InstallDebugSessionAck;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.InstallDebugSessionRequest;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.InstallState;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.StopByClientIdAck;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.StopByClientIdRequest;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.StopDebugSessionAck;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.cluster.v1.StopDebugSessionRequest;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.session.DebugSession;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.session.DebugSessionRegistry;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.session.ExecutionRecord;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.session.Granularity;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.session.InstallOutcome;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.session.Sample;
+import org.apache.skywalking.oap.server.admin.dsl.debugging.session.SessionLimits;
+import org.apache.skywalking.oap.server.core.classloader.Catalog;
+import org.apache.skywalking.oap.server.core.dsldebug.RuleKey;
+
+/**
+ * Receiver-side implementation of the cluster RPCs. Sibling of
+ * {@code RuntimeRuleClusterServiceImpl}; both register on the same
+ * cluster gRPC server (the OAP cluster bus) so peers reach each other
+ * through one transport.
+ *
+ * <h2>Workflow per RPC</h2>
+ * <pre>
+ * InstallDebugSession ─► registry.installWithId(...) ─► INSTALLED | NOT_LOCAL | REJECTED
+ * CollectDebugSamples ─► registry.find(sessionId).snapshot ─► records[] + captured + bytes
+ * StopDebugSession ─► registry.uninstall(sessionId) ─► stopped: true|false
+ * StopByClientId ─► registry.stopByClientId(clientId) ─► count + ids[]
+ * </pre>
+ *
+ * <p>Every handler is a thin adapter over the local {@link DebugSessionRegistry}
+ * so the local POST and the cluster RPC walk the same install/uninstall paths.
+ * No special-cased peer logic — once a peer binds a recorder, the local
+ * receiver-thread probes fire just like a local-installed session.
+ */
+@Slf4j
+public final class DSLDebuggingClusterServiceImpl
+ extends DSLDebuggingClusterServiceGrpc.DSLDebuggingClusterServiceImplBase {
+
+ private final DebugSessionRegistry sessionRegistry;
+ private final String selfNodeId;
+
+ public DSLDebuggingClusterServiceImpl(final DebugSessionRegistry sessionRegistry,
+ final String selfNodeId) {
+ this.sessionRegistry = sessionRegistry;
+ this.selfNodeId = selfNodeId == null ? "" : selfNodeId;
+ }
+
+ @Override
+ public void installDebugSession(final InstallDebugSessionRequest request,
+ final StreamObserver<InstallDebugSessionAck> responseObserver) {
+ final InstallDebugSessionAck.Builder ack = InstallDebugSessionAck.newBuilder()
+ .setNodeId(selfNodeId);
+ try {
+ final RuleKey ruleKey = toRuleKey(request.getRuleKey());
+ final SessionLimits limits = toLimits(request.getLimits());
+ final InstallOutcome outcome = sessionRegistry.installWithId(
+ request.getSessionId(), ruleKey, request.getClientId(), limits);
+ switch (outcome.getStatus()) {
+ case INSTALLED:
+ ack.setState(InstallState.INSTALLED);
... 12407 lines suppressed ...