(skywalking-banyandb) branch main updated: docs: KTM documentation update(workload semantics and metric interpretation) (#903)

[wu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

wusheng pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/skywalking-banyandb.git


The following commit(s) were added to refs/heads/main by this push:
     new 452668a2 docs: KTM documentation update(workload semantics and metric interpretation) (#903)
452668a2 is described below

commit 452668a23f7bde3ba09b1d64a0f50b615ce53e59
Author: Lihan Zhou <13...@users.noreply.github.com>
AuthorDate: Sat Dec 20 18:13:15 2025 -0800

    docs: KTM documentation update(workload semantics and metric interpretation) (#903)
---
 docs/design/ktm.md                 |  69 ++++----
 docs/menu.yml                      |   2 +
 docs/operation/fodc/ktm_metrics.md | 327 +++++++++++++++++++++++++++++++++++++
 3 files changed, 370 insertions(+), 28 deletions(-)

diff --git a/docs/design/ktm.md b/docs/design/ktm.md
index acf678b6..047c6de4 100644
--- a/docs/design/ktm.md
+++ b/docs/design/ktm.md
@@ -3,7 +3,7 @@
 
 ## Overview
 
-Kernel Telemetry Module (KTM) is an optional, modular kernel observability component embedded inside the BanyanDB First Occurrence Data Collection (FODC) sidecar. The first built-in module is an eBPF-based I/O monitor ("iomonitor") that focuses on page cache behavior, fadvise() effectiveness, and memory pressure signals and their impact on BanyanDB performance. KTM is not a standalone agent or network-facing service; it runs as a sub-component of the FODC sidecar ("black box") and expose [...]
+Kernel Telemetry Module (KTM) is an optional, modular kernel observability component embedded inside the BanyanDB First Occurrence Data Collection (FODC) sidecar. The first built-in module is an eBPF-based I/O monitor ("iomonitor") that focuses on page cache behavior, fadvise() effectiveness, and memory pressure signals and their impact on BanyanDB performance. KTM is not a standalone agent or network-facing service; it runs as a sub-component of the FODC sidecar ("black box") and expose [...]
 
 
 ## Architecture
@@ -50,60 +50,63 @@ Notes:
 - Focus: page cache add/delete, fadvise() calls, I/O counters, and memory reclaim signals.
 - Attachment points: stable tracepoints where possible; fentry/fexit preferred on newer kernels.
 - Data path: kernel events -> BPF maps (monotonic counters) -> userspace collector -> exporters.
-- Scoping: Fixed to the single, co-located BanyanDB process within the same container/pod.
+- Scoping: Fixed to the single, co-located BanyanDB process within the same container/pod, using cgroup membership first and a configurable comm-prefix fallback (default `banyand`).
 
 
 ## Metrics Model and Collection Strategy
 
-- Counters in BPF maps are monotonic and are not cleared by the userspace collector (NoCleanup).
+- Counters in BPF maps are monotonic and are not cleared by the userspace collector.
 - Collection and push interval: 10 seconds by default.
 - KTM periodically pushes collected metrics into the FODC Flight Recorder through a Go-native interface at the configured interval (default 10s). The push interval is exported through the `collector.interval` configuration option. The Flight Recorder is responsible for any subsequent export, persistence, or diagnostics workflows.
-- Downstream systems (for example, FODC Discovery Proxy or higher-level exporters) should derive rates using `rate()`/`irate()` or equivalents; we avoid windowed counters and map resets to preserve counter semantics.
+- Downstream systems derive rates (for example, Prometheus/PromQL `rate()`/`irate()`); FODC/KTM only provides raw counters and does not compute rates internally. We avoid windowed counters and map resets to preserve counter semantics.
 - int64 overflow is not a practical concern for our use cases; we accept long-lived monotonic growth.
+- KTM exports only raw counters; any ratios/percentages are derived upstream (see FODC operations/overview for exporter behavior).
 
 Configuration surface (current):
 - `collector.interval`: Controls the periodic push interval for metrics to Flight Recorder. Defaults to 10s.
-- `collector.enable_cgroup_filter`, `collector.enable_mntns_filter`: default on when in sidecar mode; can be toggled.
-- `collector.target_pid`/`collector.target_comm`: optional helpers for discovering scoping targets.
-- `collector.target_comm_regex`: process matcher regular expression used during target discovery (matches `/proc/<pid>/comm` and/or executable basename). Defaults to `banyand`.
-- Cleanup strategy is effectively `no_cleanup` by design intent; clear-after-read logic is deprecated for production metrics.
+- `collector.ebpf.cgroup_path` (optional): absolute or `/sys/fs/cgroup`-relative path to the BanyanDB cgroup v2; if unset, KTM autodetects by scanning `/proc/*/comm` for the configured prefix (default `banyand`).
+- `collector.ebpf.comm_prefix` (optional): comm name prefix used for process discovery and fallback filtering; defaults to `banyand`.
+- Target discovery heuristic: match `/proc/<pid>/comm` against the configured prefix (default `banyand`) to locate BanyanDB and derive its cgroup; this also serves as the runtime fallback if the cgroup filter is unset.
+- Cleanup strategy is monotonic counters only; downstream derives rates. KTM does not clear BPF maps during collection.
 - Configuration is applied via the FODC sidecar; KTM does not define its own standalone process-level configuration surface.
 
 
 ## Scoping and Filtering
 
 - Scoping is not optional; KTM is designed exclusively to monitor the single BanyanDB process it is co-located with in a sidecar deployment.
-- The target process is identified at startup, and eBPF programs are instructed to filter events to only that process.
-- Primary filtering mechanism: cgroup v2. This ensures all events originate from the correct container. PID and mount namespace filters are used as supplementary checks.
+- The target container is identified at startup; eBPF programs filter events by cgroup membership first. If the cgroup filter is absent or misses, a configurable comm-prefix match (default `banyand`) is used as a narrow fallback.
 - The design intentionally avoids multi-process or node-level (DaemonSet) monitoring to keep the implementation simple and overhead minimal.
 
 ### Target Process Discovery (Pod / VM)
 
-KTM needs to resolve the single “target” BanyanDB process before enabling filters and attaching eBPF programs. In both Kubernetes pods and VM/bare-metal deployments, KTM uses a **process matcher** driven by a configurable regular expression (`collector.target_comm_regex`, default `banyand`).
+KTM needs to resolve the single “target” BanyanDB process before enabling filters and attaching eBPF programs. In both Kubernetes pods and VM/bare-metal deployments, KTM uses a **process matcher** based on a configurable comm-prefix match (default `banyand`).
 
 #### Kubernetes Pod (sidecar)
 
 Preconditions:
-- The pod should be configured with `shareProcessNamespace: true` so the monitor sidecar can see the target container’s `/proc` entries.
-- The monitor container should have cgroup v2 mounted (typically at `/sys/fs/cgroup`).
+- The pod must be configured with `shareProcessNamespace: true` so the monitor sidecar can see the target container’s `/proc` entries.
+- cgroup v2 mounted (typically at `/sys/fs/cgroup`) to enable the primary cgroup filter.
+- If the target process cannot be discovered (for example, `shareProcessNamespace` is off), KTM logs a warning and keeps periodically checking; once the target process appears, KTM enables the module automatically.
 
 Discovery flow (high level):
 - Scan `/proc` for candidate processes.
-- For each PID, read `/proc/<pid>/comm` (and/or the executable basename) and match it against `collector.target_comm_regex`.
-- Once matched, read `/proc/<pid>/cgroup` to obtain the target’s cgroup path/identity, then enable cgroup filtering so only events from that container/process are counted.
+- For each PID, read `/proc/<pid>/comm` and match it against the configured comm prefix (default `banyand`) (or an explicitly provided cgroup path).
+- Once matched, derive the target cgroup from `/proc/<pid>/cgroup` (cgroup v2) and program the eBPF cgroup filter. The comm match remains as runtime fallback if the cgroup check does not fire.
+- If no matching process is found at startup, KTM continues periodic probing and activates once it is found.
 
 #### VM / bare metal
 
 Discovery flow (high level):
 - Scan `/proc` for candidate processes.
-- Match `/proc/<pid>/comm` (and/or executable basename) against `collector.target_comm_regex` (default `banyand`).
-- Use PID (and optionally cgroup/mount namespace filters if available) to scope kernel events to the selected process.
+- Match `/proc/<pid>/comm` against the configured prefix (default `banyand`).
+- Use the discovered PID to derive cgroup v2 path and program the filter; keep the comm match as runtime fallback if cgroup filtering is unavailable.
+- If no matching process is found at startup, KTM continues periodic probing and activates once it is found.
 
 ### Scoping Semantics
 
 - The BPF maps use a single-slot structure (e.g., a BPF array map with a single entry) to store global monotonic counters for the target process.
 - This approach eliminates the need for per-pid hash maps, key eviction logic, and complexities related to tracking multiple processes.
-- All kernel events are filtered by the target process's identity (via its cgroup ID and PID) before any counters are updated in the BPF map.
+- All kernel events are filtered by the target container’s cgroup ID when available; if the cgroup filter misses (for example, map not populated), a comm-prefix match (configurable; default `banyand`) is used before any counters are updated.
 
 Example (YAML):
 ```yaml
@@ -111,8 +114,11 @@ collector:
   interval: 10s
   modules:
     - iomonitor
-  enable_cgroup_filter: true
-  enable_mntns_filter: true
+  ebpf:
+    # Optional: absolute or /sys/fs/cgroup-relative path to the BanyanDB cgroup v2.
+    # cgroup_path: /sys/fs/cgroup/<banyandb-cgroup>
+    # Optional: comm name prefix used for process discovery and fallback filtering.
+    # comm_prefix: banyand
 ```
 
 
@@ -141,6 +147,9 @@ Prefix: metrics are currently emitted under the `ktm_` namespace to reflect thei
   - `ktm_cache_read_attempts_total`
   - `ktm_cache_misses_total`
   - `ktm_page_cache_adds_total`
+- I/O latency (syscall-level)
+  - `ktm_sys_read_latency_seconds` (histogram family: exposes `_bucket`, `_count`, `_sum`)
+  - `ktm_sys_pread_latency_seconds` (histogram family: exposes `_bucket`, `_count`, `_sum`)
 - fadvise()
   - `ktm_fadvise_calls_total`
   - `ktm_fadvise_advice_total{advice="..."}`
@@ -148,11 +157,9 @@ Prefix: metrics are currently emitted under the `ktm_` namespace to reflect thei
 - Memory
   - `ktm_memory_lru_pages_scanned_total`
   - `ktm_memory_lru_pages_reclaimed_total`
-  - `ktm_memory_reclaim_efficiency_percent`
   - `ktm_memory_direct_reclaim_processes`
 
-Semantics: all counters are monotonic; use Prometheus functions for rates/derivatives; no map clearing between scrapes.
-
+Semantics: all counters are monotonic; latency metrics are exported as Prometheus histograms (`_bucket`, `_count`, `_sum`); use Prometheus functions for rates/derivatives; no map clearing between scrapes. KTM does not emit ratio/percentage metrics; derive them upstream.
 
 ## Safety & Overhead Boundary
 
@@ -164,6 +171,8 @@ Loading and managing eBPF programs requires elevated privileges. The FODC sideca
 - `CAP_BPF`: Allows loading, attaching, and managing eBPF programs and maps. This is the preferred, more restrictive capability.
 - `CAP_SYS_ADMIN`: A broader capability that also grants permission to perform eBPF operations. It may be required on older kernels where `CAP_BPF` is not fully supported.
 
+Operational prerequisites and observability: see `docs/operation/fodc/ktm_metrics.md`.
+
 The sidecar should be configured with the minimal set of capabilities required for its operation to adhere to the principle of least privilege.
 
 ## Failure Modes
@@ -179,15 +188,19 @@ This approach ensures that a failure within the observability module does not im
 
 ## Restart Semantics
 
-On sidecar restart, BPF maps are recreated and all counters reset to zero. Downstream systems (e.g., Prometheus via FODC integrations) should treat this as a new counter lifecycle and continue deriving rates/derivatives normally.
+- On sidecar restart, BPF maps are recreated and all counters reset to zero. Downstream systems (e.g., Prometheus via FODC integrations) should treat this as a new counter lifecycle and continue deriving rates/derivatives normally.
+- If BanyanDB restarts (PID changes), the cgroup filter continues to match as long as the container does not change; the comm fallback still matches the configured prefix (default `banyand`).
+- If the pod/container is recreated (cgroup path changes), KTM re-runs target discovery, re-programs the cgroup filter, and starts counters from zero; metrics from the old container are discarded without reconciliation.
+- KTM performs a lightweight health check during collection to ensure the cgroup filter is still populated; if it is missing (for example, container crash/restart), KTM re-detects and re-programs the filter automatically.
 
 ## Kernel Attachment Points (Current)
 
-- `ksys_fadvise64_64` → fentry/fexit (preferred) or syscall tracepoints with kprobe fallback.
-- Page cache add/remove → `filemap_get_read_batch` and `mm_filemap_add_to_page_cache` tracepoints, with kprobe fallbacks.
-- Memory reclaim → `mm_vmscan_lru_shrink_inactive` and `mm_vmscan_direct_reclaim_begin` tracepoints.
+- `sys_enter_read`, `sys_exit_read`, `sys_enter_pread64`, `sys_exit_pread64` (syscall-level I/O latency).
+- `mm_filemap_add_to_page_cache`, `filemap_get_read_batch` (page cache add/churn).
+- `ksys_fadvise64_64` (fadvise policy actions; fentry/fexit preferred).
+- `mm_vmscan_lru_shrink_inactive`, `mm_vmscan_direct_reclaim_begin` (memory reclaim/pressure).
 
 ## Limitations
 
 - Page cache–only perspective: direct I/O that bypasses the cache is not observed.
-- Kernel-only visibility: no userspace spans, SQL parsing, or CPU profiling.
\ No newline at end of file
+- Kernel-only visibility: no userspace spans, SQL parsing, or CPU profiling.
diff --git a/docs/menu.yml b/docs/menu.yml
index d3e13a48..ee95f8ea 100644
--- a/docs/menu.yml
+++ b/docs/menu.yml
@@ -153,6 +153,8 @@ catalog:
         catalog:
           - name: "Overview"
             path: "/operation/fodc/overview"
+          - name: "KTM Metrics"
+            path: "/operation/fodc/ktm_metrics"
       - name: "Property Background Repair"
         path: "/concept/property-repair"
       - name: "Benchmark"
diff --git a/docs/operation/fodc/ktm_metrics.md b/docs/operation/fodc/ktm_metrics.md
new file mode 100644
index 00000000..fc1117ab
--- /dev/null
+++ b/docs/operation/fodc/ktm_metrics.md
@@ -0,0 +1,327 @@
+# KTM Metrics — Semantics & Workload Interpretation
+
+This document defines the **semantic meaning** of kernel-level metrics collected by the
+Kernel Telemetry Module (KTM) under different BanyanDB workloads.
+
+It serves as the **authoritative interpretation guide** for:
+- First Occurrence Data Capture (FODC)
+- Automated analysis and reporting by LLM agents
+- Self-healing and tuning recommendations
+
+This document does **not** describe kernel attachment points or implementation details.
+Those are covered separately in the KTM design document.
+
+---
+
+## 1. Scope and Non-Goals
+
+### In Scope
+- Interpreting kernel metrics in the context of **LSM-style read + compaction workloads**
+- Distinguishing **benign background activity** from **user-visible read-path impact**
+- Providing **actionable, explainable signals** for automated analysis
+
+### Out of Scope
+- Device-level I/O profiling or per-disk attribution
+- SLA-grade performance accounting
+- Precise block-layer root cause isolation  
+
+SLA-grade performance accounting is explicitly out of scope because
+eBPF-based sampling and histogram bucketing introduce statistical
+approximation, and kernel-level telemetry cannot capture application-
+or network-level queuing delays.  
+
+KTM focuses on **user-visible impact first**, followed by kernel-side explanations.
+
+---
+
+## 2. Core Metrics Overview
+
+### 2.1 Read / Pread Syscall Latency (Histogram)
+
+**Metric Type**
+- Histogram (bucketed latency)
+- Collected at syscall entry/exit for `read` and `pread64`
+
+**Semantic Meaning**
+This metric represents the **time BanyanDB threads spend blocked in the read/pread syscall path**.
+
+It is the **primary impact signal** in KTM.
+
+**Key Rule**
+> If syscall-level read/pread latency does **not** increase, the situation is **not considered an incident**, regardless of background cache or reclaim activity.
+
+**Why Histogram**
+- Captures long-tail latency (p95 / p99) reliably
+- More representative of user experience than averages
+- Suitable for LLM-based reasoning and reporting
+
+---
+
+### 2.2 fadvise Policy Actions
+
+**Metric Type**
+- Counter
+
+**Semantic Meaning**
+Records **explicit page cache eviction hints** issued by BanyanDB.
+
+This metric represents **policy intent**, not impact.
+
+**Interpretation Notes**
+- fadvise activity alone is not an anomaly
+- Must be correlated with read/pread latency to assess impact
+
+---
+
+### 2.3 Page Cache Add / Fill Activity
+
+**Metric Type**
+- Counter
+
+**Semantic Meaning**
+Represents pages being added to the OS page cache due to:
+- Read misses
+- Sequential scans
+- Compaction activity
+
+High page cache add rates are **expected** under LSM workloads.
+
+**Note**
+Page cache add activity does not necessarily imply disk I/O or cache miss.
+It may increase due to readahead, sequential scans, or compaction reads,
+and should be treated as a **correlated signal**, not a causal indicator,
+unless accompanied by read/pread latency degradation.
+
+---
+
+### 2.4 Memory Reclaim and Pressure Signals
+
+**Metrics**
+- LRU shrink activity
+- Direct reclaim entry events
+
+**Semantic Meaning**
+Indicates **kernel memory pressure** that may destabilize page cache residency.
+
+These metrics act as **root-cause hints**, not incident triggers.
+
+---
+
+## 3. Interpretation Principles
+
+### 3.1 Impact-First Gating
+
+All incident detection and analysis is gated on:
+
+> **Syscall-level read/pread latency histogram**
+
+This refers to the combined read/pread syscall latency histograms.
+
+Other metrics are used **only to explain why latency increased**, not to decide whether an incident occurred.
+
+---
+
+### 3.2 Cache Churn Is Not an Incident
+
+High values of:
+- page cache add
+- reclaim
+- background scans
+
+are **normal** under LSM-style workloads and **must not** be treated as incidents unless they result in read/pread latency degradation.
+
+---
+
+## 4. Workload Semantics
+
+This section defines canonical workload patterns and how KTM metrics should be interpreted.
+
+---
+
+> **Global Rule — Latency-Gated Evaluation**
+>
+> All workload patterns below are evaluated **only after syscall-level
+> read/pread latency degradation has been detected** (e.g., p95/p99 bucket shift).
+> Kernel signals such as page cache activity, reclaim, or fadvise **must not**
+> be interpreted as incident triggers on their own.
+
+---
+
+### Workload 1 — Sequential Read / Background Compaction (Benign)
+
+**Typical Signals**
+- `page_cache_add ↑`
+- `lru_shrink ↑` (optional)
+- `read/pread syscall latency stable`
+
+**Interpretation**
+Sequential scans and compaction naturally introduce cache churn.
+As long as read/pread latency remains stable, this workload is benign.
+
+**Operational Decision**
+- Do not trigger FODC
+- No self-healing action required
+
+---
+
+### Workload 2 — High Page Cache Pressure, Foreground Sustained
+
+**Typical Signals**
+- `page_cache_add ↑`
+- `lru_shrink ↑`
+- occasional `direct_reclaim`
+- `read/pread syscall latency stable`
+
+**Interpretation**
+System memory pressure exists, but foreground reads are not impacted.
+This indicates a tight but stable operating point.
+
+**Operational Decision**
+- No incident
+- Monitor trends only
+
+---
+
+### Workload 3 — Aggressive Cache Eviction or Reclaim Impact
+
+**Typical Signals**
+- `fadvise_calls ↑` or early reclaim activity
+- `page_cache_add ↑` (repeated refills)
+- `read/pread syscall latency ↑` (long-tail buckets appear)
+
+**Interpretation**
+Hot pages are evicted too aggressively, causing read amplification.
+Foreground reads are directly impacted.
+
+**Operational Decision**
+- Trigger FODC
+- Recommend tuning eviction thresholds or rate-limiting background activity
+
+**Discriminator**
+Eviction-driven degradation is typically characterized by:
+- Elevated `fadvise` activity
+- Repeated page cache refills
+- Read latency degradation **without sustained compaction throughput
+  or disk I/O saturation**
+
+- **Query pattern signal** (optional): continuously scanning an extensive time range.
+
+This pattern indicates policy-induced cache churn rather than workload contention.
+These discriminator signals are typically sourced from DB-level or system-level
+metrics outside KTM.
+
+---
+
+### Workload 4 — I/O Contention or Cold Data Access
+
+**Typical Signals**
+- `page_cache_add ↑` (due to compaction OR new data reads)
+- `read/pread syscall latency ↑`
+- reclaim may or may not be present
+
+**Interpretation**
+Latency degradation is caused by:
+1. **Resource Contention**: Compaction threads competing with foreground reads for disk I/O.
+2. **Cold Data Access**: The active working set exceeds resident memory, forcing frequent OS page cache misses (synchronous disk reads).
+
+**Operational Decision**
+- Trigger FODC
+- Suggest reducing compaction concurrency
+- If compaction is idle but latency remains high, consider scaling up memory (Capacity Planning).
+
+**Discriminator**
+This pattern is characterized by elevated read/pread syscall latency **without** the explicit eviction signals of W3 (fadvise) or the system-wide pressure of W5 (reclaim).
+It indicates that the system is physically bound by I/O limits due to contention or capacity cache misses.
+
+---
+
+### Workload 5 — OS Memory Pressure–Driven Cache Drop
+
+**Typical Signals**
+- `direct_reclaim ↑`
+- `lru_shrink ↑`
+- `read/pread syscall latency ↑`
+- `fadvise` may be absent
+
+**Interpretation**
+Cache eviction is driven by OS memory pressure rather than DB policy.
+Foreground reads stall due to synchronous reclaim.
+
+**Operational Decision**
+- Trigger FODC
+- Recommend adjusting memory limits or reducing background memory usage
+
+---
+
+## 5. Excluded Signals and Rationale
+
+### 5.1 Page Fault Metrics
+
+BanyanDB primarily uses `read()` with page cache access rather than mmap-based I/O.
+Major and minor page faults do not reliably represent read-path stalls and are therefore excluded from impact detection.
+
+### 5.2 Block Layer Latency
+
+Block-layer completion context does not reliably map to BanyanDB threads in containerized environments.
+Syscall-level latency already captures user-visible impact and is used as the primary signal.
+
+Block-layer metrics may be added later as an optional enhancement.
+
+---
+
+## 6. Summary
+
+KTM identifies read-path incidents by:
+1. Gating on **syscall-level read/pread latency histograms**
+2. Explaining impact using:
+    - eviction policy actions (fadvise)
+    - page cache behavior
+    - memory pressure signals
+
+This separation ensures:
+- Low false positives
+- Clear causality
+- Actionable and explainable self-healing decisions
+
+## 7. Decision Flow Overview
+```mermaid
+graph TD
+    Start([Start: Metric Analysis]) --> CheckLat{Read/Pread Syscall\nLatency Increased?}
+
+    %% Primary Gating Rule
+    CheckLat -- No --> Benign[Benign State\nNo User Impact]
+    CheckLat -- Yes --> Incident[Incident Detected\nTrigger FODC]
+
+    %% Benign Analysis
+    Benign --> CheckPressure{Pressure Signals\nPresent?}
+    CheckPressure -- Yes --> W2[W2: Stable State]
+    CheckPressure -- No --> W1[W1: Background Scan/Compaction]
+
+    %% Incident Analysis (Root Cause)
+    Incident --> CheckFadvise{High fadvise\ncalls?}
+    
+    %% Branch: Policy
+    CheckFadvise -- Yes --> W3[W3: Policy-Driven Eviction\nAssociated with aggressive DONTNEED (policy signal)]
+    
+    %% Branch: Kernel/OS
+    CheckFadvise -- No --> CheckReclaim{Direct Reclaim / \nLRU Shrink?}
+    
+    %% Branch: Pressure
+    CheckReclaim -- Yes --> W5[W5: OS Memory Pressure\nCause: Sync Reclaim]
+    
+    %% Branch: Contention
+    CheckReclaim -- No --> W4[W4: I/O Contention / Cold Read\nCause: Compaction or Working Set > RAM]
+
+    %% Styling
+    style CheckLat fill:#f9f,stroke:#333,stroke-width:2px
+    style Incident fill:#f00,stroke:#333,stroke-width:2px,color:#fff
+    style Benign fill:#9f9,stroke:#333,stroke-width:2px
+```
+
+---
+
+## 8. Operational Prerequisites and Observability
+
+- BTF availability and bpffs mounted are expected for fentry/fexit and map pinning where used.
+- Kernel versions must support the chosen tracepoints/fentry paths; kprobe fallbacks apply otherwise.
+- On failure to load/attach, KTM logs an error and disables itself (see Failure Modes in the design document).