You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "rawwar (via GitHub)" <gi...@apache.org> on 2025/09/24 08:47:35 UTC
[I] Add REST API metrics (latency, QPS, errors) [airflow]
rawwar opened a new issue, #56034:
URL: https://github.com/apache/airflow/issues/56034
### Description
Currently, Airflow’s REST API does not emit any metrics. It would be valuable to have basic API metrics for observability.
### Use case/motivation
- Track API performance (p95/p99 latency).
- Monitor usage patterns (QPS per endpoint).
- Alert on failures (error spikes).
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add REST API metrics (latency, QPS, errors) [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on issue #56034:
URL: https://github.com/apache/airflow/issues/56034#issuecomment-5587470177
@jasperjonkhans There is already an open PR for this close to being merged.
I would suggest to find another issue to work on :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add REST API metrics (latency, QPS, errors) [airflow]
Posted by "SruthiSathya (via GitHub)" <gi...@apache.org>.
SruthiSathya commented on issue #56034:
URL: https://github.com/apache/airflow/issues/56034#issuecomment-3976487739
Hello, if this issue is not assigned, can I work on it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add REST API metrics (latency, QPS, errors) [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on issue #56034:
URL: https://github.com/apache/airflow/issues/56034#issuecomment-3950536585
This is a very interesting feature to get. Having telemetry for the public/ui API would be great. Eventually we can discuss adding this to the execution_api too.
We just need to agree on an instrumentation tool that fits well in the airflow ecosystem to get started and it should be pretty straight forward for a basic implementation.
Jed mentioned https://github.com/trallnag/prometheus-fastapi-instrumentator. There might be other options around.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add REST API metrics (latency, QPS, errors) [airflow]
Posted by "henry3260 (via GitHub)" <gi...@apache.org>.
henry3260 commented on issue #56034:
URL: https://github.com/apache/airflow/issues/56034#issuecomment-5451362144
Scope note, since the metrics that ship differ from the three listed in the description.
#64523 emits two metrics for `/api/v2` and `/ui` requests:
- `http_requests_total` (counter)
- `http_request_duration_milliseconds` (timer)
both tagged with `method`, `route` (the FastAPI route template) and `status_family` (`2xx`/`4xx`/`5xx`).
That still covers the three use cases here:
- **latency (p95/p99)** — `http_request_duration_milliseconds`, broken down by route
- **QPS per endpoint** — `http_requests_total`, broken down by route
- **error spikes** — `http_requests_total` filtered on `status_family="5xx"`, rather than
a dedicated errors metric
The separate errors counter was dropped on purpose: it would duplicate series that `status_family` already distinguishes, and keeping the tag set small was the main constraint on this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add REST API metrics (latency, QPS, errors) [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on issue #56034:
URL: https://github.com/apache/airflow/issues/56034#issuecomment-4155733818
@henry3260 Yes I think it's a good consideration. Feel free to give it a shot if you'd like I'd be glad to review a PR or a draft in that sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add REST API metrics (latency, QPS, errors) [airflow]
Posted by "henry3260 (via GitHub)" <gi...@apache.org>.
henry3260 commented on issue #56034:
URL: https://github.com/apache/airflow/issues/56034#issuecomment-4148580151
Hi @pierrejeambrun, would it make sense to use Airflow's existing stats/observability framework rather than adding a third-party library here?
That would keep the API server aligned with the rest of Airflow's metrics approach and avoid introducing a separate path just for API-server metrics. Since the library also notes some performance cost, it seems better to start with the built-in Airflow stats abstraction if possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add REST API metrics (latency, QPS, errors) [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun closed issue #56034: Add REST API metrics (latency, QPS, errors)
URL: https://github.com/apache/airflow/issues/56034
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add REST API metrics (latency, QPS, errors) [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on issue #56034:
URL: https://github.com/apache/airflow/issues/56034#issuecomment-4006329338
@SruthiSathya Please, feel free to work on it and open a PR, you do not have to be assigned to work on a particular issue. :)
We recently updated our guidelines and we are not assigning issues anymore as per our policy,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add REST API metrics (latency, QPS, errors) [airflow]
Posted by "KatalKavya96 (via GitHub)" <gi...@apache.org>.
KatalKavya96 commented on issue #56034:
URL: https://github.com/apache/airflow/issues/56034#issuecomment-3327691407
@rawwar I would like to work on this issue. Could you please assign it to me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add REST API metrics (latency, QPS, errors) [airflow]
Posted by "rawwar (via GitHub)" <gi...@apache.org>.
rawwar commented on issue #56034:
URL: https://github.com/apache/airflow/issues/56034#issuecomment-3327700697
> [@rawwar](https://github.com/rawwar) I would like to work on this issue. Could you please assign it to me?
I actually created this issue to request comments. At this time, I am not sure if this should be implemented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add REST API metrics (latency, QPS, errors) [airflow]
Posted by "jasperjonkhans (via GitHub)" <gi...@apache.org>.
jasperjonkhans commented on issue #56034:
URL: https://github.com/apache/airflow/issues/56034#issuecomment-5463721436
Hey, I would like to give this issue a try! :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org