You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "songzhendong (via GitHub)" <gi...@apache.org> on 2026/06/05 02:22:59 UTC

[PR] Add Apache Airflow monitoring layer (SWIP-7) [skywalking]

songzhendong opened a new pull request, #13891:
URL: https://github.com/apache/skywalking/pull/13891

   ### Add Apache Airflow monitoring via native OpenTelemetry metrics (SWIP-7)
   - [x] If this is non-trivial feature, paste the links/URLs to the design doc.
     - SWIP-7: https://github.com/apache/skywalking/blob/master/docs/en/swip/SWIP-7.md (included in this PR)
   - [x] Update the documentation to include this new feature.
     - `docs/en/setup/backend/backend-airflow-monitoring.md`, `docs/en/swip/SWIP-7.md`, hierarchy docs
   - [x] Tests(including UT, IT, E2E) are added to verify the new feature.
     - MAL data tests (`airflow-service` / `airflow-instance`)
     - Mock OTLP e2e: 30 checks (`test/e2e-v2/cases/airflow/e2e.yaml`)
     - Real Celery cluster e2e: 26 checks (`test/e2e-v2/cases/airflow/e2e-cluster.yaml`)
   - [ ] If it's UI related, attach the screenshots below.
     - Horizon UI dashboards ship separately in `apache/skywalking-horizon-ui` (Workflow Scheduler / Airflow layer)
   
   - [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #<issue number>.
   - [x] Update the [\CHANGES\ log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).
   
   ## Summary
   
   - New \AIRFLOW\ layer with Service (cluster) and Instance (host) dimensions
   - MAL rules under \otel-rules/airflow/\ for Airflow native OTel metrics
   - \hierarchy-definition.yml\ links \AIRFLOW\ to \K8S_SERVICE\`n- CI: \Airflow\ + \Airflow Cluster\ jobs in \.github/workflows/skywalking.yaml\`n
   ## Test plan
   
   - [x] \./mvnw checkstyle:check\`n- [x] Local mock e2e (30/30)
   - [x] Local cluster e2e (26/26)
   - [ ] CI \Airflow\ + \Airflow Cluster\ matrix jobs


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Apache Airflow monitoring layer (SWIP-7) [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #13891:
URL: https://github.com/apache/skywalking/pull/13891#issuecomment-4631482030

   **`dataset.*` vs `asset.*` — the asset metrics only resolve on Airflow 2.x; on 3.x they'll be empty**
   
   The asset rules read the Airflow **2.x** metric names, but the setup doc targets 3.x ("Example environment variables for Airflow 3.x"). I checked the Airflow source for both lines (`v2-10-stable` and `main`):
   
   | SkyWalking metric | Reads source | Airflow 2.10 emits | Airflow 3.x (main) emits |
   |---|---|---|---|
   | `asset_updates` | `airflow_dataset_updates` | `Stats.incr("dataset.updates")` — `datasets/manager.py:149` | `stats.incr("asset.updates")` — `assets/manager.py:400` |
   | `asset_triggered_dagruns` | `airflow_dataset_triggered_dagruns` | `Stats.incr("dataset.triggered_dagruns")` — `scheduler_job_runner.py:1494` | `stats.incr("asset.triggered_dagruns")` — `scheduler_job_runner.py:2454` |
   | `asset_orphaned` | `airflow_dataset_orphaned` | `Stats.gauge("dataset.orphaned")` — `scheduler_job_runner.py:2200` | `stats.gauge("asset.orphaned")` — `scheduler_job_runner.py:3378` |
   
   This is the AIP-74/75 **Dataset → Asset** rename. In Airflow 3.x there are no `dataset.*` Stats calls left, so OAP receives `airflow_asset_*` and the current rules (filtering `airflow_dataset_*`) collect nothing → the three `asset_*` panels stay empty on 3.x. The other 25 metrics (scheduler / executor / pool / triggerer / triggers / dag_processing) are byte-identical across 2.x and 3.x, so only these three are affected. (The e2e cluster pins 2.10.5, which is why it passes today.)
   
   Not a blocker — just a version-targeting mismatch. Options:
   1. **Doc-only:** state that the asset/dataset metrics are collected for Airflow **2.x** (`dataset.*`) and that 3.x `asset.*` support is pending.
   2. **Support both:** add parallel rules reading `airflow_asset_*` alongside the `airflow_dataset_*` ones (the SkyWalking-side metric is already named `asset_*`, so it lines up) — then the layer works on 2.x and 3.x.
   
   Either way, worth making the docs + e2e target consistent. Happy to push a suggestion for whichever direction you prefer.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Apache Airflow monitoring layer (SWIP-7) [skywalking]

Posted by "songzhendong (via GitHub)" <gi...@apache.org>.
songzhendong commented on PR #13891:
URL: https://github.com/apache/skywalking/pull/13891#issuecomment-4627574650

   Horizon UI dashboards: https://github.com/apache/skywalking-horizon-ui/pull/42


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Apache Airflow monitoring layer (SWIP-7) [skywalking]

Posted by "songzhendong (via GitHub)" <gi...@apache.org>.
songzhendong commented on PR #13891:
URL: https://github.com/apache/skywalking/pull/13891#issuecomment-4655282594

   Thank you for your suggestions on the solution.
   
   SWIP-7 was drafted a long time ago. This monitoring feature was originally developed for Airflow 2.x, which exposes metrics in the `airflow.dataset.*` format. By the time we finished development and updated the deployment documents, we had set Airflow 3.x as the targeted version. However, we failed to revalidate the MAL rules and E2E test data against the new release, which resulted in this version mismatch.
   
   
   
   ---
   
   ### Changes in this commit
   
   #### 1. Align version and MAL rules for Airflow 3.x
   - **MAL rules**: Asset metric collection now uses `airflow_asset_*` (Airflow 3.x native OTel names) instead of `airflow.dataset.*` (Airflow 2.x).
   - **Documentation**: `backend-airflow-monitoring.md` and `SWIP-7.md` now clearly state that this feature targets Airflow 3.x. Metrics prefixed with `airflow.dataset.*` from Airflow 2.x are no longer collected.
   - **Test data**: All mocked OTLP replay fixtures use `airflow.asset.*`. The cluster E2E image is pinned to Airflow 3.0.6.
   
   #### 2. Additional improvements
   - **E2E layout**: Test cases are split into `mock/` and `cluster/`, following the layout used by other monitoring modules (e.g. Flink):
     - `mock/`: 29 checks — full OTLP JSON replay covering all 27 SWIP-7 metrics.
     - `cluster/`: 16 checks — native scheduler/triggerer smoke tests on Airflow 3.0.6.
   - **CI / local parity**: Custom shell verifiers (`verify-mock-e2e.sh`, `verify-cluster-e2e.sh`, etc.) are removed. Validation now uses the standard `e2e run` flow in both local environments and CI (`.github/workflows/skywalking.yaml`).
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Apache Airflow monitoring layer (SWIP-7) [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng merged PR #13891:
URL: https://github.com/apache/skywalking/pull/13891


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Apache Airflow monitoring layer (SWIP-7) [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #13891:
URL: https://github.com/apache/skywalking/pull/13891#discussion_r3362560315


##########
docs/en/setup/backend/backend-airflow-monitoring.md:
##########
@@ -0,0 +1,239 @@
+# Airflow monitoring
+
+## Airflow metrics via native OpenTelemetry
+
+SkyWalking receives Airflow metrics through Airflow's native OpenTelemetry exporter and the
+[OpenTelemetry receiver](opentelemetry-receiver.md), then aggregates them with
+[MAL](../../concepts-and-designs/mal.md).
+
+## Data flow
+
+1. Enable OpenTelemetry metrics in Airflow (`pip install 'apache-airflow[otel]'`, `otel_on = True`
+   or standard `OTEL_EXPORTER_OTLP_*` environment variables).
+2. Airflow **pushes** OTLP metrics to OpenTelemetry Collector.
+3. OpenTelemetry Collector forwards metrics to SkyWalking OAP via OTLP gRPC exporter.
+4. OAP applies MAL rules under `otel-rules/airflow/` and stores Service / Instance entities on
+   `Layer: AIRFLOW`.
+
+```mermaid
+graph LR;
+  Airflow("Airflow") --> Collector("OTel Collector")
+  Collector --> OAP("SkyWalking OAP")
+  OAP --> UI("Horizon UI")
+```
+
+In the Horizon UI, Airflow appears under the **Workflow Scheduler** menu group.
+
+## Setup
+
+### 1. Enable Airflow OpenTelemetry metrics
+
+Install the OTel extra and enable metrics export. See the
+[Airflow metrics documentation](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html).
+
+Example environment variables for Airflow 3.x:
+
+```bash
+pip install 'apache-airflow[otel]'
+
+export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
+export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
+export OTEL_RESOURCE_ATTRIBUTES=cluster=prod-airflow
+```
+
+`cluster` is required so SkyWalking can name the Airflow Service (`airflow::prod-airflow`). You
+can also inject it with a Collector `resource` processor.

Review Comment:
   The default works, but the **hard requirement** behind it isn't stated, and it's the most likely silent-failure misconfiguration. Suggest making it explicit:
   
   ```suggestion
   `cluster` is required so SkyWalking can name the Airflow Service (`airflow::prod-airflow`). You
   can also inject it with a Collector `resource` processor.
   
   > **Required — `service.name` must be `Airflow`:** OAP maps the OTLP resource `service.name` to the
   > `job_name` tag, and every MAL rule under `otel-rules/airflow/` filters on `job_name == 'Airflow'`
   > (case-sensitive). Airflow's default OTel service name is `Airflow` (the `[metrics] otel_service`
   > default in both 2.x and 3.x), so a default install works out of the box. If you override it via
   > `OTEL_SERVICE_NAME`, set it to exactly `Airflow`, and do **not** put a custom `job_name` in
   > `OTEL_RESOURCE_ATTRIBUTES` — an explicit `job_name` takes precedence over the `service.name`
   > fallback. Otherwise OAP silently drops every Airflow metric and the AIRFLOW layer stays empty.
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Apache Airflow monitoring layer (SWIP-7) [skywalking]

Posted by "songzhendong (via GitHub)" <gi...@apache.org>.
songzhendong commented on PR #13891:
URL: https://github.com/apache/skywalking/pull/13891#issuecomment-4628793423

   I'm investigating and resolving this issue, please wait for a moment.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Apache Airflow monitoring layer (SWIP-7) [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #13891:
URL: https://github.com/apache/skywalking/pull/13891#issuecomment-4627811482

   **Question: do the 7 `pool_*` e2e checks actually pass locally?**
   
   The description lists "mock e2e (30/30)" and "cluster e2e (26/26)", but I think the `pool_*` checks can't match their expected file as written — and I'd like to confirm I'm not missing something about the setup before CI runs.
   
   The pool rules keep `pool_name` as a label that is **not** an entity dimension:
   
   - `otel-rules/airflow/airflow-service.yaml`: `airflow_pool_queued_slots.sum(['cluster', 'pool_name'])` with scope `.service(['cluster'], Layer.AIRFLOW)`
   - `otel-rules/airflow/airflow-instance.yaml`: `airflow_pool_open_slots.sum(['cluster', 'host_name', 'pool_name'])` with scope `.instance(['cluster'], ['host_name'], Layer.AIRFLOW)`
   
   So `pool_name` survives as a **metric label**, and MQE returns `metric.labels: [{key: pool_name, value: default_pool}]`. But all 7 pool queries (3 service + 4 instance) in **both** `airflow-cases.yaml` and `airflow-cluster-cases.yaml` point at `expected/metrics-has-value.yml`, whose template asserts `metric.labels: []`.
   
   To check this rather than guess, I ran `skywalking-infra-e2e`'s own `verifier.Verify()` against synthetic swctl output:
   
   | Expected template | Actual result | Verdict |
   |---|---|---|
   | generic (`labels: []`) | labeled (`pool_name=default_pool`) | **MISMATCH → FAIL** |
   | generic (`labels: []`) | unlabeled | MATCH |
   | label-asserting (`- key: pool_name`) | labeled (`pool_name=default_pool`) | MATCH |
   
   This is the same single-vs-labeled split other layers already handle with a dedicated template — e.g. ActiveMQ `metrics-has-value-label-serviceinstanceid.yml`, ClickHouse `metrics-has-value-label.yml`, Flink `metrics-has-value-jobManager-node-label.yml`.
   
   Questions:
   
   1. Did the `meter_airflow_pool_*` / `meter_airflow_instance_pool_*` cases pass in your local `e2e run`? If so, could you share the `swctl metrics exec` output for one of them — does it carry a `pool_name` label?
   2. If they do carry the label, would you add an `expected/metrics-has-value-label-poolname.yml` (asserting `- key: pool_name` / `value: {{ notEmpty .value }}`) and repoint the 7 pool queries in both case files? Keeping `pool_name` in `.sum(...)` preserves the per-pool breakdown, so I'd suggest the label-template route rather than dropping the label.
   
   Happy to be corrected if there's something in the setup I've overlooked.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Apache Airflow monitoring layer (SWIP-7) [skywalking]

Posted by "songzhendong (via GitHub)" <gi...@apache.org>.
songzhendong commented on PR #13891:
URL: https://github.com/apache/skywalking/pull/13891#issuecomment-4629769376

   Root cause identified. Thanks for the detailed review — you were right about the pool expected mismatch.Before this fix, we had not run the official infra-e2e path (e2e run) locally. We were still unfamiliar with how SkyWalking e2e is wired in CI (e2e.yaml + -cases.yaml + expected/ + infra-e2e verifier), and overlooked using infra-e2e as the local validation gate. Instead, we used custom bash verify scripts (verify-mock-e2e.sh / verify-cluster-e2e.sh) that only check whether swctl metrics exec returns TIME_SERIES_VALUES with a non-null numeric value — they do not compare output against the expected/.yml templates. The earlier “mock 30/30” / “cluster 25/25” numbers came from those scripts, not from e2e run, so the label mismatch was missed locally but caught by CI and your analysis.
   Fix (e1dd30de)
   Added expected/metrics-has-value-label-poolname.yml (asserts pool_name label, same pattern as ActiveMQ).
   Repointed pool queries in airflow-cases.yaml (7) and airflow-cluster-cases.yaml (6; cluster still omits unstable pool_queued_slots).
   Verification after fix
   We ran full infra-e2e on Linux/WSL2 (same path as CI): mock 30/30, cluster 25/25 passed. We will use e2e run as the local gate going forward.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Apache Airflow monitoring layer (SWIP-7) [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #13891:
URL: https://github.com/apache/skywalking/pull/13891#issuecomment-4627626080

   Please fix the CI, license header. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Apache Airflow monitoring layer (SWIP-7) [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #13891:
URL: https://github.com/apache/skywalking/pull/13891#issuecomment-4631482548

   Thanks for the quick turnaround, and no worries — the `e2e run` path is easy to miss locally.
   
   Re-checked the fix on `e1dd30debd`: the new `metrics-has-value-label-poolname.yml` asserts `pool_name` correctly, and all 7 mock + 6 cluster pool checks are repointed to it (no check left on the `labels: []` template, and no non-pool check moved by mistake). I also ran your exact template through infra-e2e's `verifier.Verify()` against a realistic labeled result and it matches. CI now confirms it end-to-end — both **Airflow** and **Airflow Cluster** jobs are green. 👍
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org