You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "eanikindfi (via GitHub)" <gi...@apache.org> on 2024/10/28 09:20:18 UTC
[I] Invalid metric values for completed tasks task.cpu_usage_percent/task.mem_usage_percent [airflow]
eanikindfi opened a new issue, #43432:
URL: https://github.com/apache/airflow/issues/43432
### Apache Airflow version
2.10.2
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
We have Airflow `(2.10.2)` in k8s cluster deployed by an official helm-chart. This helm-release contains `statsd` component, and Airflow send its metrics to `statsd`.
We use `celery-executor`, so our tasks are running inside worker-pods.
Also we have Victoriametrics release in this cluster. We scrape metrics from `statsd` with `VMScrapeConfig`.
Out of all the metrics provided by Airflow we need [2 essential ones](https://github.com/apache/airflow/blob/v2-10-stable/docs/apache-airflow/administration-and-deployment/logging-monitoring/metrics.rst#gauges):
- task.cpu_usage_percent.<dag_id>.<task_id>
- task.mem_usage_percent.<dag_id>.<task_id>
We can see lifecycles of out tasks/dags in airflow-webinterface, so we know when each task started and ended.
But if we get metric values from statsd at the moment when our task already ended (hours after the completion), we still get the `cpu_usage` and `mem_usage` for this task.
### What you think should happen instead?
According to documentation `task.cpu_usage_percent.<dag_id>.<task_id>` and `task.mem_usage_percent.<dag_id>.<task_id>` are gauges that show:
> Percentage of CPU/memory used by a task
So we assume, that id task ended at 02:15 PM, then at 02:16 PM or even later Airflow shouldn't send either `task.cpu_usage_percent.*` or `task.mem_usage_percent.*` for this task to `statsd`, right?
The essential meaning of these 2 metrics is to show how much resources in use for each task/dag at the moment, That way we can visualize dynamic of the resource-usage of Airflow or create alerting-solutions. Correct me if I'm wrong.
### How to reproduce
**Configuration**
statsd:
```yaml
statsd:
extraMappings:
- match: airflow.task.cpu_usage.*.*
name: "airflow_task_cpu_usage"
help: "Percentage of CPU used by a task"
labels:
dag_id: "$1"
task_id: "$2"
- match: airflow.task.mem_usage.*.*
name: "airflow_task_mem_usage"
help: "Percentage of memory used by a task"
labels:
dag_id: "$1"
task_id: "$2"
```
VMScrapeConfig:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMScrapeConfig
metadata:
name: airflow-service-scrape
namespace: monitoring
spec:
staticConfigs:
- targets: [airflow-statsd.airflow.svc.cluster.local:9102]
metricsPath: /metrics
scrapeInterval: 30s
scrapeTimeout: 15s
```
### Operating System
Kubernetes v1.31.0-eks
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Even though I have provided `VMScrapeConfig` configuration, there is no need to use it in test-cases for reproduction of the issue. Because we check `statsd` endpoint (on port 9102) also to see the raw metrics delivered from Airflow - it still have the same issue.
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Invalid metric values for completed tasks task.cpu_usage_percent/task.mem_usage_percent [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #43432:
URL: https://github.com/apache/airflow/issues/43432#issuecomment-2443416580
Yeah. It looks like a "recipient-only" thing when you can compare status of the task with metrics.
Let me convert it to a discussion, because it's unlikely to be solved differently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Invalid metric values for completed tasks task.cpu_usage_percent/task.mem_usage_percent [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #43432:
URL: https://github.com/apache/airflow/issues/43432#issuecomment-2441013664
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Invalid metric values for completed tasks task.cpu_usage_percent/task.mem_usage_percent [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on issue #43432:
URL: https://github.com/apache/airflow/issues/43432#issuecomment-2442561210
It's an interesting one. As it stands, the metric emits the current loads every so often, but there is no code in the task cleanup stage which triggers that to update down (is downdate a word?) to 0%. It would be an interesting puzzle. At what point should that get zeroed out? If it is done during the cleanup stage and something happens, then it's misleading because it hasn't actually finished yet.
Perhaps some kind of check for "get all the tasks which have completed since the last time these metrics were updated and zero out their resource metrics"?
I wonder if this can be handled on the user/dashboarding end, something like "if the percentages haven't been updated after a certain amount of time then display them as zero"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Invalid metric values for completed tasks task.cpu_usage_percent/task.mem_usage_percent [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #43432: Invalid metric values for completed tasks task.cpu_usage_percent/task.mem_usage_percent
URL: https://github.com/apache/airflow/issues/43432
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Invalid metric values for completed tasks task.cpu_usage_percent/task.mem_usage_percent [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #43432:
URL: https://github.com/apache/airflow/issues/43432#issuecomment-2441282413
I think, it's the nature of statsd and we cannot do much about it (but I am not 100% sure) - and we are moving away from statsd in favour of open-telemetry https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html#setup-opentelemetry.
While we are not removing statsd, it's very likely we are not going to invest any time into improving or even fixing statsd problems.
Mybe you could switch to using open-telemetry and see if you have similar problem there - this is anyhow "future" of airflow telemetry - and it has way better features than statsd (including support for open-telemetry traces in 2.10.*) - so the best course of action would be to try it out @eanikindfi - otherwise you might hope that some will look at this one, but it's unlikely statsd will get a lot of love from the side of maintainers.
cc: @howardyoo @ferruzzi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org