You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "1fanwang (via GitHub)" <gi...@apache.org> on 2026/05/12 16:28:15 UTC
[I] Per-user rate limiting on api-server REST endpoints [airflow]
1fanwang opened a new issue, #66804:
URL: https://github.com/apache/airflow/issues/66804
### Description
The REST API on api-server is unbounded by default. A single user / service account making rapid calls (a CLI in a tight loop, misconfigured automation, a busy DAG generator) can exhaust request capacity and degrade access for the rest of a multi-tenant deployment. There's no built-in mechanism to cap per-user request rate.
For login throttling there's PR #58293 (closed) and the FAB rate-limiter for the legacy webserver — but those are about brute-force auth-page protection, not general API quota.
### Use case / motivation
Multi-tenant Airflow deployments. One noisy tenant should not be able to DoS the api-server for everyone else. Today the workarounds are:
- (a) Put a rate-limiter (envoy/nginx) in front of api-server and tag by user — but the username isn't always available at the proxy layer for token-auth flows.
- (b) Cap concurrent gunicorn workers — but that caps total throughput, not per-user.
Neither is satisfying.
### Proposal
Per-user rate-limiting middleware on api-server. Sketch:
- Keyed by authenticated principal (username from JWT / session / API token).
- Separate buckets for high-traffic endpoints (`/api/v2/monitor/health`, `/api/v2/version`) vs general API.
- Service-account principals get a higher (configurable) limit.
- Backend: in-memory by default; redis if configured (matches how the FAB rate-limiter is set up).
- Config: per-endpoint or per-route-prefix limits in `airflow.cfg`.
This is substantive enough that I'd prefer to discuss the shape on the mailing list before PR. Filing the issue so the discussion has a permanent home.
### Are you willing to submit a PR?
- [X] Yes, after design feedback
### Code of Conduct
- [X] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Per-user rate limiting on api-server REST endpoints [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #66804:
URL: https://github.com/apache/airflow/issues/66804#issuecomment-4433610736
I don't think we should do it in Airflow, to be honest. Usually things like that are done on all kinds of api gateways that you put in front of whatever app you do - and I see no particular reason Airflow will do better job than those gaeways.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Per-user rate limiting on api-server REST endpoints [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #66804:
URL: https://github.com/apache/airflow/issues/66804#issuecomment-4433637323
It goes totally opposite the way we've done with Auth - whenever there is something generic, that is not serving the "core" of airflow functionality, we should think how to move it out, not to bring it in.
I think that one is hard no for me if it comes to code changes.
And if you think this is a problem:
```
(a) Put a rate-limiter (envoy/nginx) in front of api-server and tag by user — but the username isn't always available at the proxy layer for token-auth flows.
```
We should rather think of a way how to expose the necesary missing things for the gateways rather than bringing in the whole functionalities they offer (Headers and cookies are the right ways to think about it I think).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Per-user rate limiting on api-server REST endpoints [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on issue #66804:
URL: https://github.com/apache/airflow/issues/66804#issuecomment-4483362672
Fair point — agreed the rate-limiter itself belongs in the gateway, not inside Airflow. Thanks @potiuk for the quick read, and for naming the right shape (expose what the gateway needs rather than reimplement). Closing this. If it'd help in a separate scoped issue, surfacing the JWT-decoded principal in a response header so envoy / nginx can key per-user limits without cracking the token themselves looks like a clean angle to pursue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Per-user rate limiting on api-server REST endpoints [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang closed issue #66804: Per-user rate limiting on api-server REST endpoints
URL: https://github.com/apache/airflow/issues/66804
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org