You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "mike-realuptime (via GitHub)" <gi...@apache.org> on 2026/09/02 05:13:05 UTC
[PR] Add OpenTelemetry-native host and process monitoring [skywalking]
mike-realuptime opened a new pull request, #14051:
URL: https://github.com/apache/skywalking/pull/14051
## Description
Resolves #14046.
This PR adds OpenTelemetry-native host and process monitoring for Linux and Windows while preserving the existing Prometheus exporter based monitoring paths.
The implementation keeps the final SkyWalking data model and metric aggregation in OAP/MAL. The OpenTelemetry Collector is responsible for collection, normalization of source attributes, transport, and local process cardinality control.
### Host monitoring
The existing host MAL rules are extended instead of introducing parallel rules owning the same SkyWalking target metrics:
* `vm.yaml` supports both:
* Prometheus node-exporter metrics
* OpenTelemetry hostmetrics for Linux
* `windows.yaml` supports both:
* Prometheus windows-exporter metrics
* OpenTelemetry hostmetrics for Windows
Equivalent source metrics are mapped to the same canonical SkyWalking metrics only when their semantics match.
The existing node-exporter/windows-exporter monitoring paths remain supported.
### Process monitoring
This PR adds:
* `process-hostmetrics-linux.yaml`
* `process-hostmetrics-windows.yaml`
Processes are modeled as logical SkyWalking instances using a normalized process name rather than PID.
This avoids PID-driven entity churn and keeps process cardinality bounded.
Multiple operating-system processes belonging to the same normalized process group are aggregated before being sent to OAP. SkyWalking then performs the final metric modeling through MAL.
Process metrics include:
* process count
* thread count
* CPU utilization
* resident memory usage
* resident memory utilization
* open handles / file descriptors
* oldest process uptime
### OpenTelemetry Collector
The reference configuration targets OpenTelemetry Collector Contrib 0.158.0.
The Collector:
* uses the `hostmetrics` receiver
* detects `host.name`
* adds the SkyWalking job identity
* normalizes process names
* groups equivalent processes
* aggregates process metrics before OTLP export
* sends host and process metrics through OTLP to OAP
### Backward compatibility
The existing node-exporter path remains intact.
The existing `vm.yaml` rule is extended rather than replaced, so existing Linux VM metrics continue using their current canonical SkyWalking metric names.
Node-exporter-specific metrics without an equivalent OpenTelemetry hostmetric remain sourced from node-exporter.
### Validation
A new E2E case validates the OpenTelemetry hostmetrics path using OpenTelemetry Collector Contrib 0.158.0.
It verifies:
* Linux host service discovery
* memory metrics
* CPU metrics
* logical CPU count
* normalized CPU utilization
* process logical-instance discovery
* process count aggregation
* thread aggregation
* resident memory
* open handles
* oldest process uptime
The process test starts three operating-system processes with the same executable name and verifies that SkyWalking exposes one logical process instance with a process count of 3.
The E2E test passes with:
```text
12 passed
0 failed
0 skipped
```
The existing node-exporter path was also regression-tested with the unified `vm.yaml`, including successful `OS_LINUX` service discovery and values for host memory, CPU, and node-exporter-specific `tcp_alloc`.
### Scope
This PR focuses on OAP/MAL, Collector configuration, and E2E coverage.
Horizon UI dashboards for the new process metrics will be submitted separately to the SkyWalking Horizon UI repository.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenTelemetry-native host and process monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #14051:
URL: https://github.com/apache/skywalking/pull/14051#issuecomment-5643180028
Please submit your horizon side template updates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenTelemetry-native host and process monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #14051:
URL: https://github.com/apache/skywalking/pull/14051#issuecomment-5673136993
@mike-realuptime Do you have an update? We will wait for the Horizon side to be ready, then merge this one into 11.1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenTelemetry-native host and process monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #14051:
URL: https://github.com/apache/skywalking/pull/14051#discussion_r3940803932
##########
docs/en/setup/backend/otel-collector-hostmetrics-linux.yaml:
##########
@@ -0,0 +1,261 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Environment variables:
+# SW_OAP_GRPC_ADDRESS OAP OTLP/gRPC endpoint (default 127.0.0.1:11800)
+# SW_PROCESS_NAME_REGEX process executable-name regexp (default .*)
+#
+# The Collector normalizes process identity and pre-aggregates
+# operating-system processes with the same normalized process_name before
+# OTLP export.
+
+receivers:
+ hostmetrics/system:
+ collection_interval: 30s
+ scrapers:
+ cpu:
+ metrics:
+ system.cpu.utilization:
+ enabled: true
Review Comment:
**[P1] Preserve the existing Linux CPU metric scale**
Collector Contrib 0.158.0 defaults `system.cpu.utilization` to averaging across cores. The new MAL expressions treat this value as summed CPU utilization, changing the existing `meter_vm_cpu_total_percentage` and `meter_vm_cpu_average_used` semantics. Four fully busy cores must report **400%**, but this configuration produces **100%**. The new normalized metric then divides again and reports **25%** instead of **100%**.
Please retain per-core attributes or configure sum aggregation, update the corresponding E2E configuration, and assert these values on a multi-core input. This enhancement should preserve the existing metric scale.
##########
oap-server/server-starter/src/main/resources/otel-rules/windows.yaml:
##########
@@ -13,60 +13,213 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-# This will parse a textual representation of a duration. The formats
-# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS}
-# with days considered to be exactly 24 hours.
-# <p>
-# Examples:
-# <pre>
-# "PT20.345S" -- parses as "20.345 seconds"
-# "PT15M" -- parses as "15 minutes" (where a minute is 60 seconds)
-# "PT10H" -- parses as "10 hours" (where an hour is 3600 seconds)
-# "P2D" -- parses as "2 days" (where a day is 24 hours or 86400 seconds)
-# "P2DT3H4M" -- parses as "2 days, 3 hours and 4 minutes"
-# "P-6H3M" -- parses as "-6 hours and +3 minutes"
-# "-P6H3M" -- parses as "-6 hours and -3 minutes"
-# "-P-6H+3M" -- parses as "+6 hours and -3 minutes"
-# </pre>
-filter: "{ tags -> tags.job_name == 'windows-monitoring' }" # The OpenTelemetry job name
-expSuffix: service(['node_identifier_host_name'] , Layer.OS_WINDOWS)
+# Windows infrastructure MAL.
+#
+# The existing windows-exporter source and the OpenTelemetry hostmetrics source
+# intentionally use the same existing job_name "windows-monitoring". One MAL
+# file maps either input family to the canonical meter_win_* targets, avoiding
+# cross-file metric ownership collisions in OAP 11.
+filter: "{ tags -> tags.job_name == 'windows-monitoring' }"
+expSuffix: service(['node_identifier_host_name'], Layer.OS_WINDOWS)
metricPrefix: meter_win
+
metricsRules:
- #cpu windows don't expose cpu load metrics
+ # ---------------------------------------------------------------------------
+ # Canonical windows.yaml metrics, now accepting either source.
+ # ---------------------------------------------------------------------------
+
+ # CPU
- name: cpu_total_percentage
- exp: (windows_cpu_time_total * 100).tagNotEqual('mode' , 'idle').sum(['node_identifier_host_name']).rate('PT1M')
+ exp: >
+ (windows_cpu_time_total * 100)
+ .tagNotEqual('mode','idle')
+ .sum(['node_identifier_host_name'])
+ .rate('PT1M')
+ +
+ (system_cpu_time * 100)
+ .tagNotEqual('mode','idle')
+ .sum(['node_identifier_host_name'])
+ .rate('PT1M')
+
- name: cpu_average_used
- exp: (windows_cpu_time_total * 100).sum(['node_identifier_host_name' , 'mode']).rate('PT1M')
+ exp: >
+ (windows_cpu_time_total * 100)
+ .sum(['node_identifier_host_name','mode'])
+ .rate('PT1M')
+ +
+ (system_cpu_time * 100)
+ .sum(['node_identifier_host_name','mode'])
+ .rate('PT1M')
- #memory
+ # Physical memory
- name: memory_total
- exp: windows_cs_physical_memory_bytes
+ exp: >
+ windows_cs_physical_memory_bytes
+ +
+ system_memory_limit.sum(['node_identifier_host_name'])
+
- name: memory_available
- exp: windows_os_physical_memory_free_bytes
+ exp: >
+ windows_os_physical_memory_free_bytes
+ +
+ system_memory_usage
+ .tagEqual('state','free')
+ .sum(['node_identifier_host_name'])
+
- name: memory_used
- exp: windows_cs_physical_memory_bytes - windows_os_physical_memory_free_bytes
+ exp: >
+ (windows_cs_physical_memory_bytes - windows_os_physical_memory_free_bytes)
+ +
+ (
+ system_memory_limit.sum(['node_identifier_host_name'])
+ -
+ system_memory_usage.tagEqual('state','free').sum(['node_identifier_host_name'])
+ )
+
+ # Canonical virtual memory semantics.
+ # The hostmetrics branch uses Windows Performance Counters:
+ # Memory\\Commit Limit
+ # Memory\\Committed Bytes
- name: memory_virtual_memory_free
- exp: windows_os_virtual_memory_free_bytes
+ exp: >
+ windows_os_virtual_memory_free_bytes
+ +
+ (
+ system_virtual_memory_commit_limit.sum(['node_identifier_host_name'])
+ -
+ system_virtual_memory_committed.sum(['node_identifier_host_name'])
+ )
+
- name: memory_virtual_memory_total
- exp: windows_os_virtual_memory_bytes
+ exp: >
+ windows_os_virtual_memory_bytes
+ +
+ system_virtual_memory_commit_limit.sum(['node_identifier_host_name'])
+
- name: memory_virtual_memory_percentage
- exp: 100 - ((windows_os_virtual_memory_free_bytes * 100) / windows_os_virtual_memory_bytes)
+ exp: >
+ (
+ 100 - ((windows_os_virtual_memory_free_bytes * 100) / windows_os_virtual_memory_bytes)
+ )
+ +
+ (
+ system_virtual_memory_committed.sum(['node_identifier_host_name']) * 100
+ /
+ system_virtual_memory_commit_limit.sum(['node_identifier_host_name'])
+ )
- #disk
+ # Disk throughput
- name: disk_read
- exp: windows_logical_disk_read_bytes_total.sum(['node_identifier_host_name']).rate('PT1M')
+ exp: >
+ windows_logical_disk_read_bytes_total
+ .sum(['node_identifier_host_name'])
+ .rate('PT1M')
+ +
+ system_disk_io
+ .tagEqual('direction','read')
+ .sum(['node_identifier_host_name'])
+ .rate('PT1M')
+
- name: disk_written
- exp: windows_logical_disk_write_bytes_total.sum(['node_identifier_host_name']).rate('PT1M')
+ exp: >
+ windows_logical_disk_write_bytes_total
+ .sum(['node_identifier_host_name'])
+ .rate('PT1M')
+ +
+ system_disk_io
+ .tagEqual('direction','write')
+ .sum(['node_identifier_host_name'])
+ .rate('PT1M')
- #network
+ # Network throughput
- name: network_receive
- exp: windows_net_bytes_received_total.sum(['node_identifier_host_name']).irate()
+ exp: >
+ windows_net_bytes_received_total
+ .sum(['node_identifier_host_name'])
+ .irate()
+ +
+ system_network_io
+ .tagEqual('direction','receive')
+ .sum(['node_identifier_host_name'])
+ .irate()
+
- name: network_transmit
- exp: windows_net_bytes_sent_total.sum(['node_identifier_host_name']).irate()
-
-
-
-
+ exp: >
+ windows_net_bytes_sent_total
+ .sum(['node_identifier_host_name'])
+ .irate()
+ +
+ system_network_io
+ .tagEqual('direction','transmit')
+ .sum(['node_identifier_host_name'])
+ .irate()
+
+ # ---------------------------------------------------------------------------
+ # Additional hostmetrics capabilities. These are additive and do not change
+ # the existing windows-exporter metric contract.
+ # ---------------------------------------------------------------------------
+ - name: cpu_cores_num
+ exp: system_cpu_logical_count.sum(['node_identifier_host_name'])
+
+ - name: cpu_norm_percentage
+ exp: >
+ (
+ (system_cpu_time * 100)
+ .tagNotEqual('mode','idle')
+ .sum(['node_identifier_host_name'])
+ .rate('PT1M')
Review Comment:
**[P2] Avoid overlapping CPU states in the new Windows normalized metric**
Native Windows `system` CPU time already includes interrupt time. Summing every non-idle state adds `interrupt` again, inflating the new `meter_win_cpu_norm_percentage` and potentially exceeding 100%.
Please calculate this new normalized metric from non-overlapping user and system time. The existing total-CPU rule already sums all non-idle modes; this finding concerns the **new normalized metric**, rather than a regression in the existing exporter path.
##########
docs/en/setup/backend/backend-vm-monitoring.md:
##########
@@ -2,56 +2,71 @@
SkyWalking leverages Prometheus node-exporter to collect metrics data from the VMs and leverages OpenTelemetry Collector to transfer the metrics to
[OpenTelemetry receiver](opentelemetry-receiver.md) and into the [Meter System](./../../concepts-and-designs/mal.md).
VM entity as a `Service` in OAP and on the `Layer: OS_LINUX`.
-
SkyWalking also provides InfluxDB Telegraf to receive VMs' metrics data by [Telegraf receiver](./telegraf-receiver.md).
The telegraf receiver plugin receiver, process and convert the metrics, then it send converted metrics to [Meter System](./../../concepts-and-designs/mal.md).
VM entity as a `Service` in OAP and on the `Layer: OS_LINUX`.
-
## Data flow
**For OpenTelemetry receiver:**
1. The Prometheus node-exporter collects metrics data from the VMs.
2. The OpenTelemetry Collector fetches metrics from node-exporter via Prometheus Receiver and pushes metrics to the SkyWalking OAP Server via OpenTelemetry gRPC exporter.
3. The SkyWalking OAP Server parses the expression with [MAL](../../concepts-and-designs/mal.md) to filter/calculate/aggregate and store the results.
-
**For Telegraf receiver:**
1. The InfluxDB Telegraf [input plugins](https://docs.influxdata.com/telegraf/v1.24/plugins/) collects various metrics data from the VMs.
2. The cpu, mem, system, disk and diskio input plugins should be set in telegraf.conf file.
2. The InfluxDB Telegraf send `JSON` format metrics by `HTTP` messages to Telegraf Receiver, then pushes converted metrics to the SkyWalking OAP Server [Meter System](./../../concepts-and-designs/mal.md).
3. The SkyWalking OAP Server parses the expression with [MAL](../../concepts-and-designs/mal.md) to filter/calculate/aggregate ad store the results.
4. The meter_vm_cpu_average_used metrics indicates the average usage of each CPU core for telegraf receiver.
-
## Setup
**For OpenTelemetry receiver:**
1. Setup [Prometheus node-exporter](https://prometheus.io/docs/guides/node-exporter/).
2. Setup [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/). This is an example for OpenTelemetry Collector configuration [otel-collector-config.yaml](../../../../test/e2e-v2/cases/vm/prometheus-node-exporter/otel-collector-config.yaml).
3. Config SkyWalking [OpenTelemetry receiver](opentelemetry-receiver.md).
-
**For Telegraf receiver:**
1. Setup InfluxDB Telegraf's `telegraf.conf file` according to [Telegraf office document](https://docs.influxdata.com/telegraf/v1.24/).
2. Setup InfluxDB Telegraf's `telegraf.conf file` specific rules according to [Telegraf receiver document](telegraf-receiver.md).
3. Config SkyWalking [Telegraf receiver](telegraf-receiver.md).
+### Native OpenTelemetry hostmetrics (extended alternative)
+SkyWalking can also receive Linux host and process metrics directly from the OpenTelemetry Collector `hostmetrics` receiver, without Prometheus node-exporter.
+
+1. Install OpenTelemetry Collector Contrib `0.158.0` on the monitored host.
+2. Use the deployable reference configuration [otel-collector-hostmetrics-linux.yaml](otel-collector-hostmetrics-linux.yaml).
+3. Set `SW_OAP_GRPC_ADDRESS` to the SkyWalking OAP OTLP/gRPC endpoint. Optionally set `SW_PROCESS_NAME_REGEX` to restrict the process executable names to collect.
+4. Config SkyWalking [OpenTelemetry receiver](opentelemetry-receiver.md). The default OAP rule set enables both `vm` and `process-hostmetrics-linux`.
+
+The Collector normalizes host and process identity, compacts process resource metrics, aligns process datapoint timestamps to the collection window, and pre-aggregates operating-system processes with the same normalized `process_name` before OTLP export. `process-hostmetrics-linux.yaml` maps the aggregated metrics to the corresponding SkyWalking process instance.
+
## Supported Metrics
-| Monitoring Panel | Unit | Metric Name | Description | Data Source |
-|------------------------------|------|-------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------|-----------------------------------------------------|
-| CPU Usage | % | meter_vm_cpu_total_percentage | The total percentage usage of the CPU core. If there are 2 cores, the maximum usage is 200%. | Prometheus node-exporter<br />Telegraf input plugin |
-| Memory RAM Usage | MB | meter_vm_memory_used | The total RAM usage | Prometheus node-exporter<br />Telegraf input plugin |
-| Memory Swap Usage | % | meter_vm_memory_swap_percentage | The percentage usage of swap memory | Prometheus node-exporter<br />Telegraf input plugin |
-| CPU Average Used | % | meter_vm_cpu_average_used | The percentage usage of the CPU core in each mode | Prometheus node-exporter<br />Telegraf input plugin |
-| CPU Load | | meter_vm_cpu_load1<br />meter_vm_cpu_load5<br />meter_vm_cpu_load15 | The CPU 1m / 5m / 15m average load | Prometheus node-exporter<br />Telegraf input plugin |
-| Memory RAM | MB | meter_vm_memory_total<br />meter_vm_memory_available<br />meter_vm_memory_used<br />meter_vm_memory_buff_cache | The RAM statistics, including Total / Available / Used / Buff-Cache | Prometheus node-exporter<br />Telegraf input plugin |
-| Memory Swap | MB | meter_vm_memory_swap_free<br />meter_vm_memory_swap_total | Swap memory statistics, including Free / Total | Prometheus node-exporter<br />Telegraf input plugin |
-| File System Mountpoint Usage | % | meter_vm_filesystem_percentage | The percentage usage of the file system at each mount point | Prometheus node-exporter<br />Telegraf input plugin |
-| Disk R/W | KB/s | meter_vm_disk_read<br />meter_vm_disk_written | The disk read and written | Prometheus node-exporter<br />Telegraf input plugin |
-| Network Bandwidth Usage | KB/s | meter_vm_network_receive<br />meter_vm_network_transmit | The network receive and transmit | Prometheus node-exporter<br />Telegraf input plugin |
-| Network Status | | meter_vm_tcp_curr_estab<br />meter_vm_tcp_tw<br />meter_vm_tcp_alloc<br />meter_vm_sockets_used<br />meter_vm_udp_inuse | The number of TCPs established / TCP time wait / TCPs allocated / sockets in use / UDPs in use | Prometheus node-exporter<br />Telegraf input plugin |
-| Filefd Allocated | | meter_vm_filefd_allocated | The number of file descriptors allocated | Prometheus node-exporter |
+The two OpenTelemetry receiver paths do not expose exactly the same Linux kernel metrics. In particular, OpenTelemetry Collector Contrib `0.158.0` `hostmetrics` has no host-level metric equivalent to node-exporter's `node_filefd_allocated`, which is derived from `/proc/sys/fs/file-nr`. The per-process `process.open_file_descriptors` metric has different semantics and is intentionally not used as a substitute. Therefore, `meter_vm_filefd_allocated` remains available only through the node-exporter path.
+
+Likewise, `meter_vm_tcp_alloc`, `meter_vm_sockets_used`, and `meter_vm_udp_inuse` remain node-exporter-only because `hostmetrics` `system.network.connections` does not expose equivalent Linux sockstat counters. `meter_vm_tcp_curr_estab` and `meter_vm_tcp_tw` do have compatible hostmetrics equivalents.
+
+| Monitoring Panel | Unit | Metric Name | Description | node-exporter | OTel hostmetrics | Telegraf |
+|---|---|---|---|:---:|:---:|:---:|
+| CPU Usage | % | `meter_vm_cpu_total_percentage` | Total CPU usage across all logical CPUs | Yes | Yes | Yes |
+| CPU Cores | count | `meter_vm_cpu_cores_num` | Number of logical CPUs | — | Yes | — |
+| Normalized CPU Usage | % | `meter_vm_cpu_norm_percentage` | CPU usage normalized by the number of logical CPUs | — | Yes | — |
+| CPU Average Used | % | `meter_vm_cpu_average_used` | CPU usage by mode/state | Yes | Yes | Yes |
+| CPU Load | | `meter_vm_cpu_load1`<br />`meter_vm_cpu_load5`<br />`meter_vm_cpu_load15` | CPU 1m / 5m / 15m average load | Yes | Yes | Yes |
+| Memory RAM Usage | MB | `meter_vm_memory_used` | Total RAM usage | Yes | Yes | Yes |
+| Memory RAM | MB | `meter_vm_memory_total`<br />`meter_vm_memory_available`<br />`meter_vm_memory_used`<br />`meter_vm_memory_buff_cache` | RAM Total / Available / Used / Buff-Cache | Yes | Yes | Yes |
+| Memory Swap Usage | % | `meter_vm_memory_swap_percentage` | Percentage of swap memory in use | Yes | Yes | Yes |
+| Memory Swap | MB | `meter_vm_memory_swap_free`<br />`meter_vm_memory_swap_total` | Swap Free / Total | Yes | Yes | Yes |
+| File System Mountpoint Usage | % | `meter_vm_filesystem_percentage` | File-system usage at each mount point | Yes | Yes | Yes |
+| Disk R/W | KB/s | `meter_vm_disk_read`<br />`meter_vm_disk_written` | Disk read and write throughput | Yes | Yes | Yes |
+| Network Bandwidth Usage | KB/s | `meter_vm_network_receive`<br />`meter_vm_network_transmit` | Network receive and transmit throughput | Yes | Yes | Yes |
+| TCP Established / Close-Wait | count | `meter_vm_tcp_curr_estab` | TCP connections in ESTABLISHED or CLOSE-WAIT state | Yes | Yes | Yes |
+| TCP Time Wait | count | `meter_vm_tcp_tw` | TCP connections in TIME-WAIT state | Yes | Yes | Yes |
+| TCP Allocated | count | `meter_vm_tcp_alloc` | Allocated TCP sockets | Yes | No | Yes |
+| Sockets Used | count | `meter_vm_sockets_used` | Kernel sockets currently in use | Yes | No | Yes |
+| UDP In Use | count | `meter_vm_udp_inuse` | UDP sockets currently in use | Yes | No | Yes |
+| Filefd Allocated | count | `meter_vm_filefd_allocated` | Host-level allocated file descriptors from Linux `/proc/sys/fs/file-nr` | Yes | No | — |
+| Network Connections | count | `meter_vm_network_connections` | TCP connections grouped by protocol and state | — | Yes | — |
Review Comment:
**[P3] Remove or implement the advertised network-connections metric**
The supported-metrics table lists `meter_vm_network_connections`, but no corresponding MAL rule is included. Users following this documentation cannot query that metric. Please remove the entry or provide the rule that produces it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenTelemetry-native host and process monitoring [skywalking]
Posted by "mike-realuptime (via GitHub)" <gi...@apache.org>.
mike-realuptime commented on PR #14051:
URL: https://github.com/apache/skywalking/pull/14051#issuecomment-5804815414
The corresponding Horizon UI dashboard changes are now available in:
https://github.com/apache/skywalking-horizon-ui/pull/171
This completes the UI side of the OpenTelemetry host/process monitoring work introduced by this PR.
The Horizon changes add:
- updated OS_LINUX and OS_WINDOWS host dashboards;
- normalized CPU and logical CPU metrics from the new MAL rules;
- OpenTelemetry hostmetrics-specific memory, filesystem, network, handle/file-descriptor, and pagefile metrics;
- Top Process views using the new process metrics;
- Linux and Windows process instance dashboards;
- updated i18n overlays for all bundled locales.
I validated the dashboards end-to-end against a live OAP containing the MAL changes from this PR, using OpenTelemetry Collector Contrib 0.158.0 with real Linux and Windows host/process telemetry.
Horizon validation is clean: type-check, 3206 unit tests, lint, build, template validation, i18n validation, license check, and manual live-telemetry validation all pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenTelemetry-native host and process monitoring [skywalking]
Posted by "mike-realuptime (via GitHub)" <gi...@apache.org>.
mike-realuptime commented on PR #14051:
URL: https://github.com/apache/skywalking/pull/14051#issuecomment-5640311897
Thanks for the follow-up. I reproduced the issue and confirmed both remaining points.
I have updated the Windows normalized CPU expression to combine the `user` and `system` modes before calling `rate()`, using the suggested `tagMatch('mode','^(user|system)$')` expression. The production and E2E MAL copies are synchronized.
I also moved the `user/system/interrupt/idle` regression samples into the fixture `input` section and updated the expected values. The fixture now explicitly verifies that `interrupt` is present in the source data but excluded from normalized CPU calculation.
I reran the MAL tests and the relevant validation successfully.
Thanks again for the precise reproduction and explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenTelemetry-native host and process monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #14051:
URL: https://github.com/apache/skywalking/pull/14051#issuecomment-5522186201
Thanks for the contribution. I reviewed the current head and reproduced the Collector path with `otelcol-contrib:0.158.0`. I think the following should be addressed before merge:
1. **Fix the MAL unit-test data.** All current unit-test jobs fail with 12 `EMPTY` results. This is not an existing exporter-output regression: the new VM/Windows hostmetrics-only expressions have no matching input in `vm.data.yaml` and `windows.data.yaml`. Please extend those fixtures, and add runtime fixtures for the two new process MAL files (which are currently compile-only).
2. **The advertised Collector-side process aggregation does not occur.** Every scrape still exports three `process.count=1` datapoints for the three `sleep` processes, rather than one `process.count=3` datapoint. The datapoints have slightly different timestamps, and `aggregate_on_attributes` includes the timestamp in its grouping key. MAL then sums the three values within the report, so the final OAP value is correctly `3` and repeated reports are averaged rather than accumulated. Please either remove the ineffective Collector aggregation and document that MAL performs the value aggregation, or implement actual pre-OTLP aggregation and assert the raw Collector output.
3. **The native Windows path does not match the MAL inputs.** `windows.yaml` filters/groups `system_cpu_time` by `mode`, but hostmetrics emits `state`; idle is therefore included and the per-mode result collapses. `process-hostmetrics-windows.yaml` expects `process_open_handles`, while native Windows hostmetrics emits `process.handles` (`process_handles` in MAL). No Windows normalization configuration or Windows test is included.
4. **The process MAL rules are not enabled by the default OAP configuration.** `enabledOtelMetricsRules` contains `vm` and `windows`, but neither `process-hostmetrics-linux` nor `process-hostmetrics-windows`. The E2E masks this by explicitly enabling the Linux process rule. Please either enable them by default or document the required opt-in configuration.
5. **The existing node-exporter TopN-with-attributes assertion is regressed.** The rewritten `cpu_load1` expression removes the previous `decorate({ me -> me.attr0 = me.layer.name() })`, while the E2E still queries `top_n(..., attr0='OS_LINUX')`. Restore the decoration or intentionally update the contract and assertion.
6. **Several mappings described as equivalent have different semantics:**
- `tcp_curr_estab` maps only `state=established`, whereas `Tcp_CurrEstab` includes ESTABLISHED and CLOSE-WAIT.
- OTel CPU states `interrupt` and `wait` are copied unchanged, while the existing node-exporter label values are `irq` and `iowait`.
- OTel/gopsutil cached memory includes `SReclaimable`, while `node_memory_Cached_bytes` does not, so `memory_buff_cache` changes meaning depending on its source.
7. **`process.count` inherits the `{threads}` unit.** `copy_metric` copies the descriptor from `process.threads`; the exact runtime output reports `process.count Unit: {threads}`. Please set the unit explicitly to `{process}`.
8. **The production/reference configuration and Windows coverage are missing.** The only Collector configuration added is explicitly Linux E2E configuration, despite the PR advertising Linux and Windows production support. Please add deployable reference documentation/configuration and Windows OTLP fixture coverage.
The `process_memory_utilization * 100` expression is intentional fixed-point scaling for UI precision, so I do not consider it an output bug. Its comment should describe the basis-point scaling, and the corresponding UI expression should divide by `100`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenTelemetry-native host and process monitoring [skywalking]
Posted by "mike-realuptime (via GitHub)" <gi...@apache.org>.
mike-realuptime commented on PR #14051:
URL: https://github.com/apache/skywalking/pull/14051#issuecomment-5673816858
Ok. templates are ready as per my taste on it, but I still have work to do on the e2e testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenTelemetry-native host and process monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #14051:
URL: https://github.com/apache/skywalking/pull/14051#issuecomment-5577923385
Thanks for the updates. I re-reviewed `63a7ed3f52d05a6b4ce6bb768ff8bcde5b12a91b`. The Linux CPU aggregation correction and documentation cleanup look good. I have two remaining requests, with the input data and verified outputs below.
**1. Combine user and system before calling `rate()` for Windows normalized CPU.**
The [current expression](https://github.com/apache/skywalking/blob/63a7ed3f52d05a6b4ce6bb768ff8bcde5b12a91b/oap-server/server-starter/src/main/resources/otel-rules/windows.yaml#L164-L181) calls `rate('PT1M')` separately for user and system, after each branch removes `mode` with `.sum(['node_identifier_host_name'])`. Both calls consequently use the same counter-history key: metric owner, sample name, and remaining host label.
All CPU modes arrive together in one `system_cpu_time` sample family within each export. The collision occurs between the two `rate()` calls inside the same MAL evaluation.
I verified this with a complete Collector Contrib 0.158.0 export: 589 requests and 52,499 datapoints. The capture used the Linux reference configuration with 2-second collection intervals and a `sleep` process allowlist. I then applied the PR's Windows host normalization, including `state` to `mode`, preserving all values, timestamps, and request boundaries. This is real Linux collection replayed against the Windows MAL; I did not capture native Windows telemetry.
Each full request was decoded by the actual `OpenTelemetryMetricRequestProcessor`, then evaluated through compiled MAL and the production `Expression.run` wrapper. This replay stopped before storage/query processing.
Here is the CPU portion of two complete exports, exactly 60 seconds apart. CPU time values are cumulative seconds; all eight modes were included together in each export, along with the other metric families.
| Input | First export | Export 60 seconds later |
| --- | ---: | ---: |
| `mode=user` | 4.99 | 5.53 |
| `mode=system` | 11.79 | 12.97 |
| `mode=idle` | 5520.20 | 6116.68 |
| `mode=interrupt` | 0 | 0 |
| `mode=nice` | 0 | 0 |
| `mode=softirq` | 1.47 | 1.80 |
| `mode=steal` | 0 | 0 |
| `mode=wait` | 3.14 | 3.14 |
| `system_cpu_logical_count` | 10 | 10 |
The expected normalized user + system percentage is:
```text
((5.53 - 4.99) + (12.97 - 11.79)) / 60 / 10 * 100
= 0.2866666667%
```
| MAL | Actual compiled output |
| --- | ---: |
| Current PR: separate user/system rates | **1.4200000000%** |
| Proposed: combine user/system, then one rate | **0.2866666667%** |
In the current expression, the system branch uses the user baseline `4.99`, producing `(12.97 - 4.99) / 60 * 100 = 13.3`. Adding the user branch's `0.9` and dividing by 10 gives the incorrect `1.42%`.
Please use:
```yaml
- name: cpu_norm_percentage
exp: >
(system_cpu_time * 100)
.tagMatch('mode','^(user|system)$')
.sum(['node_identifier_host_name'])
.rate('PT1M')
/
system_cpu_logical_count.sum(['node_identifier_host_name'])
```
The regex works in compiled MAL and selects exactly `user` and `system`. Please synchronize the production and E2E rule copies.
**2. Move the intended CPU regression samples into `input` and update the assertions.**
The new [`system_cpu_time` block](https://github.com/apache/skywalking/blob/63a7ed3f52d05a6b4ce6bb768ff8bcde5b12a91b/oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/windows.data.yaml#L294-L309), containing user `20`, system `12`, interrupt `8`, and idle `60`, is under `expected`. It is therefore neither supplied as input nor checked as a rule output. The actual input still contains only user `20` and idle `80`, with 4 logical CPUs.
This explains why the existing fixture passes without detecting the issue:
| Test input | Expected normalized output | Current MAL | Proposed MAL |
| --- | ---: | ---: | ---: |
| Existing fixture, using its normal counter priming | 125 | 125 — PASS | 125 — PASS |
| Two complete real exports shown above | 0.2866666667 | 1.42 — FAIL | 0.2866666667 — PASS |
Both comparisons were executed with compiled MAL. Please move the misplaced samples into `input`, update the affected expectations, and include a regression with user and system present together. Include interrupt data to verify that it is excluded.
Validation: the backend build and all 1,437 local MAL module tests passed. The full exported-data replay evaluated each CPU expression 295 times; 294 outputs differed, with both expressions returning zero for the initial observation without counter history. Replaying the two complete exports above separately also reproduced the failure and verified the proposed result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenTelemetry-native host and process monitoring [skywalking]
Posted by "mike-realuptime (via GitHub)" <gi...@apache.org>.
mike-realuptime commented on PR #14051:
URL: https://github.com/apache/skywalking/pull/14051#issuecomment-5544573507
Thank you very much for the detailed review.
I have addressed the points you raised and pushed an updated version of the PR.
This review has been a particularly valuable learning experience for me. Several of the issues you pointed out helped me better understand both SkyWalking's metric contracts and the OpenTelemetry Collector processing model. I sincerely appreciate the time you spent reproducing the setup and explaining the problems so precisely.
The changes are summarized below:
1. Extended the Linux VM and Windows MAL fixtures with the missing hostmetrics inputs and added runtime fixtures for both new process MAL rules.
2. Implemented real Collector-side logical process aggregation before OTLP export. The E2E now verifies the raw post-processor Collector output and confirms that three `sleep` processes become exactly one `process.count=3` datapoint before OAP/MAL processing.
3. Added the required Windows normalization:
- `system.cpu.time`: `state` -> `mode`
- `process.handles` -> `process.open_handles`
4. Added `process-hostmetrics-linux` and `process-hostmetrics-windows` to the default `enabledOtelMetricsRules`.
5. Restored the historical `cpu_load1` decoration so the TopN `attr0=OS_LINUX` contract is preserved.
6. Corrected the semantic mappings for:
- `Tcp_CurrEstab`
- CPU `interrupt` -> `irq`
- CPU `wait` -> `iowait`
- cached-memory semantics
7. `process.count` is now explicitly created with unit `{process}`.
8. Added deployable Linux and Windows Collector reference configurations and Windows OTLP host/process E2E coverage.
The final validation is green, including:
- MAL runtime tests
- backend build
- Linux hostmetrics E2E
- Linux process E2E
- raw Collector aggregation/normalization assertions
- Windows OTLP host/process E2E coverage
- existing node-exporter regression E2E
Thank you again for the thorough review and for the opportunity to improve both the implementation and my understanding of SkyWalking's architecture.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenTelemetry-native host and process monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng merged PR #14051:
URL: https://github.com/apache/skywalking/pull/14051
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenTelemetry-native host and process monitoring [skywalking]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #14051:
URL: https://github.com/apache/skywalking/pull/14051#discussion_r3911044272
##########
oap-server/server-starter/src/main/resources/otel-rules/vm.yaml:
##########
@@ -1,4 +1,3 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
Review Comment:
The Apache 2.0 license header is incomplete: the leading "Licensed to the Apache Software Foundation (ASF) under one or more" line is missing, which breaks the standard header format used elsewhere in the repo (e.g., windows.yaml).
##########
test/e2e-v2/cases/vm/prometheus-node-exporter/otel-rules/vm.yaml:
##########
@@ -1,4 +1,3 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
Review Comment:
The Apache 2.0 license header is incomplete: the leading "Licensed to the Apache Software Foundation (ASF) under one or more" line is missing. YAML files in this repo are expected to carry the full standard header.
##########
test/e2e-v2/cases/vm/otel-hostmetrics/otel-rules/vm.yaml:
##########
@@ -0,0 +1,257 @@
+# contributor license agreements. See the NOTICE file distributed with
Review Comment:
The Apache 2.0 license header is incomplete: the leading "Licensed to the Apache Software Foundation (ASF) under one or more" line is missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenTelemetry-native host and process monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #14051:
URL: https://github.com/apache/skywalking/pull/14051#issuecomment-5633947524
@mike-realuptime Any update about this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org