You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "songzhendong (via GitHub)" <gi...@apache.org> on 2024/04/04 08:00:37 UTC
[PR] Add Apache Airflow Monitoring [skywalking]
songzhendong opened a new pull request, #12072:
URL: https://github.com/apache/skywalking/pull/12072
### Add Apache Airflow Monitoring
- [x] Update the documentation to include this new feature.
- Documentation has been updated in the `docs` directory.
- [x] Tests(including UT, IT, E2E) are added to verify the new feature.
- Unit tests and integration tests have been added under `test` directory.
- [x] UI the screenshots below.
- [x] #12071
- [x] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#discussion_r1554831337
##########
docs/en/swip/readme.md:
##########
@@ -71,7 +71,7 @@ All accepted and proposed SWIPs can be found in [here](https://github.com/apache
Next SWIP Number: 6
Review Comment:
```suggestion
Next SWIP Number: 8
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#discussion_r1554587580
##########
docs/menu.yml:
##########
@@ -188,6 +188,8 @@ catalog:
path: "/en/setup/service-agent/virtual-cache"
- name: "Virtual MQ"
path: "/en/setup/service-agent/virtual-mq"
+ - name: "Airflow"
+ path: "/en/setup/backend/backend-airflow-monitoring"
Review Comment:
This should be a new catalog, `Workflow Scheduler`. Like you did for the UI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wankai123 (via GitHub)" <gi...@apache.org>.
wankai123 commented on code in PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#discussion_r1554797512
##########
test/e2e-v2/cases/airflow/airflow-main/docker-compose.yaml:
##########
@@ -0,0 +1,286 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: '2.1'
+
+services:
+ # SkyWalking OAP server
+ oap:
+ extends:
+ file: ../../../script/docker-compose/base-compose.yml
+ service: oap
+ ports:
+ - 12800:12800
+ entrypoint: ['sh', '-c', '/download-mysql.sh /skywalking/oap-libs && chmod 777 /skywalking/docker-entrypoint.sh && /skywalking/docker-entrypoint.sh']
+ networks:
+ - e2e
+ environment:
+ - TZ=Asia/Shanghai
+ - SW_STORAGE=mysql
+ - SW_DATA_SOURCE_USER=root
+ - SW_DATA_SOURCE_PASSWORD=password
+ - SW_JDBC_URL=jdbc:mysql://mysql:3306/swtest?allowMultiQueries=true
+ - SW_CORE_TOPN_REPORT_PERIOD=2
+ depends_on:
+ mysql:
+ condition: service_healthy
+
+ # SkyWalking UI
+ ui:
+ image: xiaodong12315/skywalking-ui:2.0
+ ports:
+ - "8082:8080"
+ networks:
+ - e2e
+ environment:
+ - SW_OAP_ADDRESS=http://oap:12800
+ - TZ=Asia/Shanghai
+ depends_on:
+ oap:
+ condition: service_healthy
+
+ # MySQL database
+ mysql:
+ image: mysql:8.0.13
+ networks:
+ - e2e
+ volumes:
+ - ./my.cnf:/etc/my.cnf
+ environment:
+ - "MYSQL_ROOT_PASSWORD=password"
+ - "MYSQL_DATABASE=swtest"
+ ports:
+ - 3306
+ healthcheck:
+ test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/3306"]
+ interval: 5s
+ timeout: 60s
+ retries: 120
+
+ # OpenTelemetry collector 1
+ otel-collector:
+ image: otel/opentelemetry-collector-contrib:0.96.0
+ networks:
+ - e2e
+ command: [ "--config=/etc/otel-collector-config.yaml" ]
+ volumes:
+ - ../airflow-exporter/otel-collector-config.yaml:/etc/otel-collector-config.yaml
+ expose:
+ - 55678
Review Comment:
you didn't expose `8889` here, how can receive metrics?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#issuecomment-2063956604
No update for a week. @songzhendong If you want to continue to make this ready to merge, please let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#discussion_r1554985214
##########
docs/en/changes/changes.md:
##########
@@ -103,7 +106,9 @@
* Enhance VNode logic and support multiple Trace IDs in span's ref.
* Add the layers filed and associate layers dashboards for the service topology nodes.
* Fix `Nginx-Instance` metrics to instance level.
-* Update tabs of the Kubernetes service page.
+* Update tabs of the Kubernetes service page.
+* Support Airflow monitoring.
+* Update tabs of the Kubernetes service page.
Review Comment:
```suggestion
* Update tabs of the Kubernetes service page.
* Support Airflow monitoring.
```
Duplicated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#discussion_r1555120308
##########
test/e2e-v2/cases/airflow/airflow-exporter/otel-collector-config.yaml:
##########
@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+receivers:
+ otlp:
+ protocols:
+ http:
+ endpoint: "0.0.0.0:8889" # Receiver configuration using the OTLP protocol over HTTP, listening Airflow on port 8889
+
+exporters:
+ otlp:
+ endpoint: "oap:11800" # Exporter configuration, sending data to the specified endpoint address
+ tls:
+ insecure: true # Disables TLS security or TLS certificate verification in the testing environment
+
+processors:
+ transform:
+ metric_statements:
+ - context: resource
+ statements: []
+ - context: metric
+ statements:
+ - set(name, ConvertCase(name, "snake")) # Transform metric names to snake case
+
+ metricstransform:
+ transforms:
+ - include: "^airflow.*" # Select metrics starting with 'airflow' for transformation
+ match_type: regexp # Use regular expression for matching metric names
+ action: update
+ operations:
+ - action: add_label
+ new_label: job_name # Add a new label 'job_name' to the selected metrics
+ new_value: "airflow-monitoring"
+ - include: "^airflow.*"
+ match_type: regexp
+ action: update
+ operations:
+ - action: add_label
+ new_label: service_instance_id # Add 'service_instance_id' label to metrics
+ new_value: "instance1"
Review Comment:
If this is impossible(if there is no value to have metadata in OTLP push), I think we should recommend using K8s sidecar as recommended deployment to add this label. In there, we could use pod name as the instance name, and k8s service name as the service name.
Meanwhile, is `host_name` representing the service name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#issuecomment-2041486644
Please recheck UI submodule, you are changing that unexpectedly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wankai123 (via GitHub)" <gi...@apache.org>.
wankai123 commented on code in PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#discussion_r1554763079
##########
test/e2e-v2/cases/airflow/airflow-main/airflow2.cfg:
##########
@@ -0,0 +1,2224 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+[core]
+# The folder where your airflow pipelines live, most likely a
+# subfolder in a code repository. This path must be absolute.
+#
+# Variable: AIRFLOW__CORE__DAGS_FOLDER
+#
+dags_folder = ./airflow/dags
Review Comment:
What's the difference between `airflow.cfg` and `airflow2.cfg`, the default configuration is too long, could we simplify them?
##########
test/e2e-v2/cases/airflow/airflow-main/docker-compose.yaml:
##########
@@ -0,0 +1,286 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: '2.1'
+
+services:
+ # SkyWalking OAP server
+ oap:
+ extends:
+ file: ../../../script/docker-compose/base-compose.yml
+ service: oap
+ ports:
+ - 12800:12800
+ entrypoint: ['sh', '-c', '/download-mysql.sh /skywalking/oap-libs && chmod 777 /skywalking/docker-entrypoint.sh && /skywalking/docker-entrypoint.sh']
+ networks:
+ - e2e
+ environment:
+ - TZ=Asia/Shanghai
+ - SW_STORAGE=mysql
+ - SW_DATA_SOURCE_USER=root
+ - SW_DATA_SOURCE_PASSWORD=password
+ - SW_JDBC_URL=jdbc:mysql://mysql:3306/swtest?allowMultiQueries=true
+ - SW_CORE_TOPN_REPORT_PERIOD=2
+ depends_on:
+ mysql:
+ condition: service_healthy
+
+ # SkyWalking UI
+ ui:
+ image: xiaodong12315/skywalking-ui:2.0
+ ports:
+ - "8082:8080"
+ networks:
+ - e2e
+ environment:
+ - SW_OAP_ADDRESS=http://oap:12800
+ - TZ=Asia/Shanghai
+ depends_on:
+ oap:
+ condition: service_healthy
+
+ # MySQL database
+ mysql:
+ image: mysql:8.0.13
+ networks:
+ - e2e
+ volumes:
+ - ./my.cnf:/etc/my.cnf
+ environment:
+ - "MYSQL_ROOT_PASSWORD=password"
+ - "MYSQL_DATABASE=swtest"
+ ports:
+ - 3306
+ healthcheck:
+ test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/3306"]
+ interval: 5s
+ timeout: 60s
+ retries: 120
+
+ # OpenTelemetry collector 1
+ otel-collector:
+ image: otel/opentelemetry-collector-contrib:0.96.0
+ networks:
+ - e2e
+ command: [ "--config=/etc/otel-collector-config.yaml" ]
+ volumes:
+ - ../airflow-exporter/otel-collector-config.yaml:/etc/otel-collector-config.yaml
+ expose:
+ - 55678
+ depends_on:
+ oap:
+ condition: service_healthy
+ environment:
+ - TZ=Asia/Shanghai
+
+ # OpenTelemetry collector 2
+ otel-collector2:
+ image: otel/opentelemetry-collector-contrib:0.96.0
+ networks:
+ - e2e
+ command: [ "--config=/etc/otel-collector-config.yaml" ]
+ volumes:
+ - ../airflow-exporter/otel-collector-config2.yaml:/etc/otel-collector-config.yaml
Review Comment:
Do we have to use two `otel-collector` with different configs? Users usually deploy only one to collect all services's metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#issuecomment-2041134149
And, you still don't resolve conflicts. Learn git more please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "songzhendong (via GitHub)" <gi...@apache.org>.
songzhendong closed pull request #12072: Add Apache Airflow Monitoring
URL: https://github.com/apache/skywalking/pull/12072
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#issuecomment-2041076127
With https://github.com/apache/skywalking/pull/12076 gets merged, you should have the expected menu.
Then, you should fix all CI tasks. UI submodule should be reverted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wankai123 (via GitHub)" <gi...@apache.org>.
wankai123 commented on code in PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#discussion_r1554793813
##########
test/e2e-v2/cases/airflow/airflow-main/docker-compose.yaml:
##########
@@ -0,0 +1,286 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: '2.1'
+
+services:
+ # SkyWalking OAP server
+ oap:
+ extends:
+ file: ../../../script/docker-compose/base-compose.yml
+ service: oap
+ ports:
+ - 12800:12800
+ entrypoint: ['sh', '-c', '/download-mysql.sh /skywalking/oap-libs && chmod 777 /skywalking/docker-entrypoint.sh && /skywalking/docker-entrypoint.sh']
+ networks:
+ - e2e
+ environment:
+ - TZ=Asia/Shanghai
+ - SW_STORAGE=mysql
Review Comment:
If MySQL is only used for OAP, it is suggested to remove it and use the default `h2` instead in order to reduce the GHA resource.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#issuecomment-2041122182
<img width="832" alt="image" src="https://github.com/apache/skywalking/assets/5441976/4d0f875e-9b81-4d03-896e-84f900808baa">
You still don't fix the conflicts, CI would not run until you fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "songzhendong (via GitHub)" <gi...@apache.org>.
songzhendong closed pull request #12072: Add Apache Airflow Monitoring
URL: https://github.com/apache/skywalking/pull/12072
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#issuecomment-2041409947
Don't close a PR, as you only update things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "songzhendong (via GitHub)" <gi...@apache.org>.
songzhendong commented on PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#issuecomment-2047994604
> FYI @songzhendong We will update the MAL implementation for #11992. Then UI side will update to support multiple-label tags as well.
>
> You will need to update this PR accordingly, most about UI dashboard setup and e2e.
get
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wankai123 (via GitHub)" <gi...@apache.org>.
wankai123 commented on code in PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#discussion_r1554796948
##########
oap-server/server-starter/src/main/resources/ui-initialized-templates/airflow/airflow-instance.json:
##########
@@ -0,0 +1,395 @@
+[
+ {
+ "id": "Airflow-Instance",
+ "configuration": {
+ "children": [
+ {
+ "x": 0,
+ "y": 0,
+ "w": 6,
+ "h": 9,
+ "i": "0",
+ "type": "Widget",
+ "graph": {
+ "type": "Bar",
+ "fontSize": 40,
+ "textAlign": "center",
+ "showUnit": true
+ },
+ "widget": {
+ "title": "Airflow Job Related"
+ },
+ "metricConfig": [
+ {
+ "label": "Job Start Num"
+ }
+ ],
+ "metricMode": "Expression",
+ "expressions": [
+ "meter_airflow_instance_job_start"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ]
+ },
+ {
+ "x": 0,
+ "y": 9,
+ "w": 6,
+ "h": 10,
+ "i": "1",
+ "type": "Widget",
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": false,
+ "showXAxis": true,
+ "showYAxis": true,
+ "showUnit": true
+ },
+ "widget": {
+ "title": "Instance Executor Related"
+ },
+ "metricMode": "Expression",
+ "metricConfig": [
+ {
+ "label": "Open Slots Nums"
+ },
+ {
+ "label": "K8s Instances Duration(ms)"
+ }
+ ],
+ "expressions": [
+ "meter_airflow_instance_executor_open_slots",
+ "meter_airflow_instance_kubernetes_executor_adopt_task_instances_duration"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES",
+ "TIME_SERIES_VALUES"
+ ]
+ },
+ {
+ "x": 12,
+ "y": 0,
+ "w": 6,
+ "h": 9,
+ "i": "2",
+ "type": "Widget",
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": false,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "title": "Scheduler Related"
+ },
+ "metricConfig": [
+ {
+ "label": "Heartbeat"
+ },
+ {
+ "label": "Critical Section Busy"
+ }
+ ],
+ "metricMode": "Expression",
+ "expressions": [
+ "meter_airflow_instance_scheduler_heartbeat(rate)",
Review Comment:
MQE doesn't support this format `xxx(rate)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wankai123 (via GitHub)" <gi...@apache.org>.
wankai123 commented on code in PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#discussion_r1554792979
##########
test/e2e-v2/cases/airflow/airflow-main/docker-compose.yaml:
##########
@@ -0,0 +1,286 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: '2.1'
+
+services:
+ # SkyWalking OAP server
+ oap:
+ extends:
+ file: ../../../script/docker-compose/base-compose.yml
+ service: oap
+ ports:
+ - 12800:12800
+ entrypoint: ['sh', '-c', '/download-mysql.sh /skywalking/oap-libs && chmod 777 /skywalking/docker-entrypoint.sh && /skywalking/docker-entrypoint.sh']
+ networks:
+ - e2e
+ environment:
+ - TZ=Asia/Shanghai
+ - SW_STORAGE=mysql
+ - SW_DATA_SOURCE_USER=root
+ - SW_DATA_SOURCE_PASSWORD=password
+ - SW_JDBC_URL=jdbc:mysql://mysql:3306/swtest?allowMultiQueries=true
+ - SW_CORE_TOPN_REPORT_PERIOD=2
+ depends_on:
+ mysql:
+ condition: service_healthy
+
+ # SkyWalking UI
+ ui:
+ image: xiaodong12315/skywalking-ui:2.0
+ ports:
+ - "8082:8080"
+ networks:
+ - e2e
+ environment:
+ - SW_OAP_ADDRESS=http://oap:12800
+ - TZ=Asia/Shanghai
+ depends_on:
+ oap:
+ condition: service_healthy
Review Comment:
```suggestion
```
We don't require SkyWalking UI for e2e in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#discussion_r1554794056
##########
docs/en/setup/backend/backend-ariflow-monitoring.md:
##########
@@ -0,0 +1,56 @@
+# Apache Airflow monitoring
+## Airflow server performance from 'OpenTelemetry'
+SkyWalking leverages Airflow for collecting metrics data. It leverages OpenTelemetry Collector to transfer the metrics to
+[OpenTelemetry receiver](opentelemetry-receiver.md) and into the [Meter System](./../../concepts-and-designs/meter.md).
+
+### Data flow
+
+1. OpenTelemetry Collector fetches metrics from Airflow via OTEL and pushes metrics to SkyWalking OAP Server via OpenTelemetry exporter.
Review Comment:
How to fetch through OTEL? OTEL protocol is a push style.
Could you clarify this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#issuecomment-2044327249
FYI @songzhendong We will update the MAL implementation for https://github.com/apache/skywalking/issues/11992. Then UI side will update to support multiple-label tags as well.
You will need to update this PR accordingly, most about UI dashboard setup and e2e.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#discussion_r1554637150
##########
docs/en/changes/changes.md:
##########
@@ -104,7 +106,10 @@
* Add the layers filed and associate layers dashboards for the service topology nodes.
* Fix `Nginx-Instance` metrics to instance level.
* Update tabs of the Kubernetes service page.
-
+* Support Airflow monitoring.
+* Add Airflow menu i18n.
+* Add Support for dragging in the trace panel.
+* Add workflow icon.
Review Comment:
```suggestion
* Add workflow icon.
```
Same here. Don't change original style.
##########
docs/en/changes/changes.md:
##########
@@ -130,5 +135,5 @@
* Remove `OpenTelemetry Exporter` support from meter doc, as this has been flagged as unmaintained on OTEL upstream.
* Add doc of one-line quick start script for different storage types.
* Add FAQ for `Why is Clickhouse or Loki or xxx not supported as a storage option?`.
-
+* Add Airflow monitoring.
Review Comment:
```suggestion
* Add Airflow monitoring docs.
```
##########
docs/en/changes/changes.md:
##########
@@ -80,7 +80,9 @@
* Support displaying the port services listen to from OAP and UI during server start.
* Refactor data-generator to support generating metrics.
* Fix `AvgHistogramPercentileFunction` legacy name.
-
+* Support Airflow monitoring.
+* Add Service Hierarchy auto matching layer relationships (upper -> lower) as following:
+ - AIRFLOW -> K8S_SERVICE
Review Comment:
```suggestion
- AIRFLOW -> K8S_SERVICE
```
You should not remove the last empty line.
##########
oap-server/server-starter/src/main/resources/hierarchy-definition.yml:
##########
@@ -103,7 +106,7 @@ layer-levels:
RABBITMQ: 2
KAFKA: 2
PULSAR: 2
-
+ AIRFLOW: 2
Review Comment:
```suggestion
AIRFLOW: 2
```
You changed many style, please revert them.
##########
docs/en/setup/backend/backend-ariflow-monitoring.md:
##########
@@ -0,0 +1,56 @@
+# Apache Airflow monitoring
+## Airflow server performance from 'OpenTelemetry'
+SkyWalking leverages Airflow for collecting metrics data. It leverages OpenTelemetry Collector to transfer the metrics to
+[OpenTelemetry receiver](opentelemetry-receiver.md) and into the [Meter System](./../../concepts-and-designs/meter.md).
+
+### Data flow
+
+1. OpenTelemetry Collector fetches metrics from Airflow via OTEL and pushes metrics to SkyWalking OAP Server via OpenTelemetry exporter.
+2. The SkyWalking OAP Server parses the expression with [MAL](../../concepts-and-designs/mal.md) to filter/calculate/aggregate and store the results.
+
+### Set up
+1. Set up [Airflow](https://airflow.apache.org/docs/apache-airflow/stable/start.html).
+2. Set up [Airflow[otel]](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html).
+3. Set up [OpenTelemetry Collector ](https://opentelemetry.io/docs/collector/getting-started/#docker). For details on Receiver in OpenTelemetry Collector, refer to [here](../../../../test/e2e-v2/cases/airflow/airflow-exporter/otel-collector-config.yaml).
+4. Config SkyWalking [OpenTelemetry receiver](opentelemetry-receiver.md).
+
+### Airflow Monitoring
+Airflow monitoring provides monitoring of the status and resources of Airflow . Airflow is cataloged as a `Layer: Airflow` `Service` in OAP.
+#### Airflow Service Supported Metrics
+| Monitoring Panel | Unit | Metric Name | Description | Data Source |
+|-----|------|-----|-----|-----|
+| Airflow Job Started | count | <job_name>_start | Number of started job | OpenTelemetry from Airflow |
+| Tasks Executable | count | scheduler.tasks.executable | Number of tasks that are ready for execution (set to queued) with respect to pool limits, DAG concurrency, executor state, and priority. | OpenTelemetry from Airflow |
+| Tasks Cleared | count | scheduler.orphaned_tasks.cleared | Number of Orphaned tasks cleared by the Scheduler | OpenTelemetry from Airflow |
+| Tasks Adopted | count | scheduler.orphaned_tasks.adopted | Number of Orphaned tasks adopted by the Scheduler | OpenTelemetry from Airflow |
+| Queued Tasks | count | executor.queued_tasks | Number of queued tasks on executor | OpenTelemetry from Airflow |
+| Pool Open Slots | count | executor.open_slots | Number of open slots on executor | OpenTelemetry from Airflow |
+| Pool Queued Slots | count | pool.queued_slots | Number of queued slots in the pool. Metric with pool_name tagging. | OpenTelemetry from Airflow |
+| Deferred Slots | count | pool.deferred_slots | Number of deferred slots in the pool. Metric with pool_name tagging. | OpenTelemetry from Airflow |
+| Scheduler Heartbeat | rate | scheduler_heartbeat | Scheduler heartbeats | OpenTelemetry from Airflow |
+| DAG File Queue Size | count | dag_processing.file_path_queue_size | Number of DAG files to be considered for the next scan | OpenTelemetry from Airflow |
+| Dataset Updates | count | dataset.updates | Number of updated datasets | OpenTelemetry from Airflow |
+### Airflow Instance Supported Metrics
Review Comment:
```suggestion
### Airflow Instance Supported Metrics
```
Is the table working without empty line?
##########
docs/en/concepts-and-designs/service-hierarchy.md:
##########
@@ -36,7 +36,7 @@ If you want to customize it according to your own needs, please refer to [Servic
| VIRTUAL_DATABASE | CLICKHOUSE | [VIRTUAL_DATABASE On CLICKHOUSE](#virtual_database-on-clickhouse) |
| PULSAR | K8S_SERVICE | [PULSAR On K8S_SERVICE](#pulsar-on-k8s_service) |
| VIRTUAL_MQ | PULSAR | [VIRTUAL_MQ On PULSAR](#virtual_mq-on-pulsar) |
-
+| AIRFLOW | K8S_SERVICE | [AIRFLOW On K8S_SERVICE](#airflow-on-k8s_service) |
Review Comment:
```suggestion
| AIRFLOW | K8S_SERVICE | [AIRFLOW On K8S_SERVICE](#airflow-on-k8s_service) |
```
No empty line could break markdown doc easily.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#issuecomment-2041083959
SWIP and relative updates are missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#discussion_r1555122490
##########
test/e2e-v2/cases/airflow/airflow-exporter/otel-collector-config.yaml:
##########
@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+receivers:
+ otlp:
+ protocols:
+ http:
+ endpoint: "0.0.0.0:8889" # Receiver configuration using the OTLP protocol over HTTP, listening Airflow on port 8889
+
+exporters:
+ otlp:
+ endpoint: "oap:11800" # Exporter configuration, sending data to the specified endpoint address
+ tls:
+ insecure: true # Disables TLS security or TLS certificate verification in the testing environment
+
+processors:
+ transform:
+ metric_statements:
+ - context: resource
+ statements: []
+ - context: metric
+ statements:
+ - set(name, ConvertCase(name, "snake")) # Transform metric names to snake case
+
+ metricstransform:
+ transforms:
+ - include: "^airflow.*" # Select metrics starting with 'airflow' for transformation
+ match_type: regexp # Use regular expression for matching metric names
+ action: update
+ operations:
+ - action: add_label
+ new_label: job_name # Add a new label 'job_name' to the selected metrics
+ new_value: "airflow-monitoring"
+ - include: "^airflow.*"
+ match_type: regexp
+ action: update
+ operations:
+ - action: add_label
+ new_label: service_instance_id # Add 'service_instance_id' label to metrics
+ new_value: "instance1"
+ - include: "^airflow.*"
+ match_type: regexp
+ action: update
+ operations:
+ - action: add_label
+ new_label: host_name # Add 'host_name' label indicating the host name
+ new_value: "airflow-webserver.airflow"
Review Comment:
If this is the service name, and hard coded instance name, I doubt the service and instance hierarchy would work.
I saw your screenshot, but that seems not match your e2e.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#discussion_r1554630917
##########
oap-server/server-starter/src/main/resources/ui-initialized-templates/menu.yaml:
##########
@@ -47,6 +47,16 @@ menus:
description: Observe the virtual message queue servers which are conjectured by language agents through various plugins.
documentLink: https://skywalking.apache.org/docs/main/next/en/setup/service-agent/virtual-mq/
i18nKey: general_service_virtual_mq
+ - title: Workflow Scheduler
+ icon: workflow_scheduler
+ description: Provide monitoring for workflow scheduling systems
+ i18nKey: workflow_scheduler
+ menus:
+ - title: Airflow
+ layer: AIRFLOW
+ description: Provide Airflow monitoring through OpenTelemetry otlp Receiver.
+ documentLink: https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-airflow-monitoring/
+ i18nKey: workflow_scheduler_airflow
Review Comment:
Same here, you should not randomly add it as the first one. It should align with the one I suggested for doc menu.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "songzhendong (via GitHub)" <gi...@apache.org>.
songzhendong commented on PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#issuecomment-2047995724
> FYI @songzhendong We will update the MAL implementation for #11992. Then UI side will update to support multiple-label tags as well.
>
> You will need to update this PR accordingly, most about UI dashboard setup and e2e.
Get
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#discussion_r1554630859
##########
docs/menu.yml:
##########
@@ -188,6 +188,10 @@ catalog:
path: "/en/setup/service-agent/virtual-cache"
- name: "Virtual MQ"
path: "/en/setup/service-agent/virtual-mq"
+ - name: "Workflow Scheduler"
Review Comment:
<img width="373" alt="image" src="https://github.com/apache/skywalking/assets/5441976/b0899a67-3425-4457-9db0-c458d13831a2">
You should put the menu item here. Rather than randomly adding into the first one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wankai123 (via GitHub)" <gi...@apache.org>.
wankai123 commented on code in PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#discussion_r1554796948
##########
oap-server/server-starter/src/main/resources/ui-initialized-templates/airflow/airflow-instance.json:
##########
@@ -0,0 +1,395 @@
+[
+ {
+ "id": "Airflow-Instance",
+ "configuration": {
+ "children": [
+ {
+ "x": 0,
+ "y": 0,
+ "w": 6,
+ "h": 9,
+ "i": "0",
+ "type": "Widget",
+ "graph": {
+ "type": "Bar",
+ "fontSize": 40,
+ "textAlign": "center",
+ "showUnit": true
+ },
+ "widget": {
+ "title": "Airflow Job Related"
+ },
+ "metricConfig": [
+ {
+ "label": "Job Start Num"
+ }
+ ],
+ "metricMode": "Expression",
+ "expressions": [
+ "meter_airflow_instance_job_start"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES"
+ ]
+ },
+ {
+ "x": 0,
+ "y": 9,
+ "w": 6,
+ "h": 10,
+ "i": "1",
+ "type": "Widget",
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": false,
+ "showXAxis": true,
+ "showYAxis": true,
+ "showUnit": true
+ },
+ "widget": {
+ "title": "Instance Executor Related"
+ },
+ "metricMode": "Expression",
+ "metricConfig": [
+ {
+ "label": "Open Slots Nums"
+ },
+ {
+ "label": "K8s Instances Duration(ms)"
+ }
+ ],
+ "expressions": [
+ "meter_airflow_instance_executor_open_slots",
+ "meter_airflow_instance_kubernetes_executor_adopt_task_instances_duration"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES",
+ "TIME_SERIES_VALUES"
+ ]
+ },
+ {
+ "x": 12,
+ "y": 0,
+ "w": 6,
+ "h": 9,
+ "i": "2",
+ "type": "Widget",
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": false,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "title": "Scheduler Related"
+ },
+ "metricConfig": [
+ {
+ "label": "Heartbeat"
+ },
+ {
+ "label": "Critical Section Busy"
+ }
+ ],
+ "metricMode": "Expression",
+ "expressions": [
+ "meter_airflow_instance_scheduler_heartbeat(rate)",
Review Comment:
MQE doesn't support this format `(rate)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wankai123 (via GitHub)" <gi...@apache.org>.
wankai123 commented on code in PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#discussion_r1554974594
##########
test/e2e-v2/cases/airflow/airflow-main/docker-compose.yaml:
##########
@@ -0,0 +1,139 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: '2.1'
+
+services:
+ # SkyWalking OAP server
+ oap:
+ extends:
+ file: ../../../script/docker-compose/base-compose.yml
+ service: oap
+ ports:
+ - 12800:12800
+ networks:
+ - e2e
+ environment:
+ - TZ=Asia/Shanghai
+ - SW_CORE_TOPN_REPORT_PERIOD=2
+ - SW_STORAGE=h2
+ # OpenTelemetry collector 1
+ otel-collector:
+ image: otel/opentelemetry-collector-contrib:0.96.0
Review Comment:
```suggestion
image: otel/opentelemetry-collector:${OTEL_COLLECTOR_VERSION}
```
##########
oap-server/server-starter/src/main/resources/ui-initialized-templates/airflow/airflow-service.json:
##########
@@ -0,0 +1,365 @@
+[
+ {
+ "id": "Airflow-Service",
+ "configuration": {
+ "children": [
+ {
+ "x": 0,
+ "y": 0,
+ "w": 24,
+ "h": 33,
+ "i": "0",
+ "type": "Tab",
+ "children": [
+ {
+ "name": "Inspections",
+ "children": [
+ {
+ "x": 0,
+ "y": 0,
+ "w": 6,
+ "h": 13,
+ "i": "1",
+ "type": "Widget",
+ "graph": {
+ "type": "Card",
+ "fontSize": 124,
+ "textAlign": "center",
+ "showUnit": true
+ },
+ "widget": {
+ "title": "Airflow Job",
+ "name": "FileOpen",
+ "tips": "This count includes the number of all instances"
+ },
+ "metricMode": "Expression",
+ "metricConfig": [
+ {
+ "label": "job Start Nums"
+ }
+ ],
+ "expressions": [
+ "latest(aggregate_labels(meter_airflow_job_start,sum))"
+ ],
+ "typesOfMQE": [
+ "SINGLE_VALUE"
+ ]
+ },
+ {
+ "x": 6,
+ "y": 0,
+ "w": 6,
+ "h": 13,
+ "i": "2",
+ "type": "Widget",
+ "graph": {
+ "type": "Line",
+ "step": false,
+ "smooth": false,
+ "showSymbol": false,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "title": "Tasks Related",
+ "tips": "This count includes the number of all instances"
+ },
+ "metricConfig": [
+ {
+ "label": "Tasks Executable"
+ },
+ {
+ "label": "Tasks Cleared"
+ },
+ {
+ "label": "Tasks Adopted"
+ },
+ {
+ "label": "Queued Tasks"
+ }
+ ],
+ "metricMode": "Expression",
+ "expressions": [
+ "aggregate_labels(meter_airflow_scheduler_tasks_executable,sum)",
+ "aggregate_labels(meter_airflow_scheduler_orphaned_tasks_cleared,sum)",
+ "aggregate_labels(meter_airflow_scheduler_orphaned_tasks_adopted,sum)",
+ "aggregate_labels(meter_airflow_executor_queued_tasks,sum)"
+ ],
+ "typesOfMQE": [
+ "TIME_SERIES_VALUES",
+ "TIME_SERIES_VALUES",
+ "TIME_SERIES_VALUES",
+ "TIME_SERIES_VALUES"
+ ]
+ },
+ {
+ "x": 12,
+ "y": 0,
+ "w": 6,
+ "h": 13,
+ "i": "3",
+ "type": "Widget",
+ "graph": {
+ "type": "Area",
+ "opacity": 0.4,
+ "showXAxis": true,
+ "showYAxis": true
+ },
+ "widget": {
+ "title": "Dag Related",
+ "tips": "This count includes the number of all instances"
+ },
+ "metricConfig": [
+ {
+ "label": "Dag File Queue Size"
+ },
+ {
+ "label": "Dag Processes"
+ }
+ ],
+ "metricMode": "Expression",
+ "expressions": [
+ "aggregate_labels(meter_airflow_dag_processing_file_path_queue_size,sum))",
+ "aggregate_labels(meter_airflow_dag_processing_processes,sum))"
Review Comment:
```suggestion
"aggregate_labels(meter_airflow_dag_processing_file_path_queue_size,sum)",
"aggregate_labels(meter_airflow_dag_processing_processes,sum)"
```
##########
test/e2e-v2/cases/airflow/airflow-main/docker-compose.yaml:
##########
@@ -0,0 +1,139 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: '2.1'
+
+services:
+ # SkyWalking OAP server
+ oap:
+ extends:
+ file: ../../../script/docker-compose/base-compose.yml
+ service: oap
+ ports:
+ - 12800:12800
+ networks:
+ - e2e
+ environment:
+ - TZ=Asia/Shanghai
+ - SW_CORE_TOPN_REPORT_PERIOD=2
+ - SW_STORAGE=h2
+ # OpenTelemetry collector 1
+ otel-collector:
+ image: otel/opentelemetry-collector-contrib:0.96.0
Review Comment:
We'd better use a unified version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wankai123 (via GitHub)" <gi...@apache.org>.
wankai123 commented on code in PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#discussion_r1554996072
##########
test/e2e-v2/cases/airflow/airflow-exporter/otel-collector-config.yaml:
##########
@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+receivers:
+ otlp:
+ protocols:
+ http:
+ endpoint: "0.0.0.0:8889" # Receiver configuration using the OTLP protocol over HTTP, listening Airflow on port 8889
+
+exporters:
+ otlp:
+ endpoint: "oap:11800" # Exporter configuration, sending data to the specified endpoint address
+ tls:
+ insecure: true # Disables TLS security or TLS certificate verification in the testing environment
+
+processors:
+ transform:
+ metric_statements:
+ - context: resource
+ statements: []
+ - context: metric
+ statements:
+ - set(name, ConvertCase(name, "snake")) # Transform metric names to snake case
+
+ metricstransform:
+ transforms:
+ - include: "^airflow.*" # Select metrics starting with 'airflow' for transformation
+ match_type: regexp # Use regular expression for matching metric names
+ action: update
+ operations:
+ - action: add_label
+ new_label: job_name # Add a new label 'job_name' to the selected metrics
+ new_value: "airflow-monitoring"
+ - include: "^airflow.*"
+ match_type: regexp
+ action: update
+ operations:
+ - action: add_label
+ new_label: service_instance_id # Add 'service_instance_id' label to metrics
+ new_value: "instance1"
Review Comment:
When airflow metrics use the push mode, do the metrics have any labels to identify which service and instance?
You add labels manually here, I'm not sure when 1 otel-collector facing 2 airflow instances, how to transform the metrics and add labels.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Apache Airflow Monitoring [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #12072:
URL: https://github.com/apache/skywalking/pull/12072#discussion_r1555121756
##########
test/e2e-v2/cases/airflow/airflow-main/e2e.yaml:
##########
@@ -0,0 +1,38 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file is used to show how to write configuration files and can be used to test.
+
+setup:
+ env: compose
Review Comment:
Please make sure whether compose is a good way to run metadata clear deployment. If it isn't, airflow instance + otel collector as sidecar deployment on k8s may be a better way.
e2e is not just for testing, you could see on the docs(you wrote), all configurations here are a kind of guidance and reference for users to deploy this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org