You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2026/05/25 22:04:58 UTC
[PR] Cap default httpx connection pool on the task-sdk Client [airflow]
potiuk opened a new pull request, #67506:
URL: https://github.com/apache/airflow/pull/67506
`Client.__init__` did not set explicit `httpx.Limits`, so each Client inherited httpx's defaults of `max_connections=100` and `max_keepalive_connections=10`. The supervisor's own [`_ensure_client`](https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L1119) capped at `max_connections=10, max_keepalive_connections=1` — so the two code paths that build a task-sdk Client diverged by an order of magnitude in their pool behaviour, with no documented reason and surprising resource use under high concurrency.
Reported as F-011 in the [`apache/tooling-agents` L3 task-sdk sweep `0920c77`](https://github.com/apache/tooling-agents/issues/24).
## Change
Standardise on a single bounded default at the Client level via `kwargs.setdefault("limits", httpx.Limits(max_keepalive_connections=5, max_connections=20))`. The supervisor's lower-cap path is unchanged (it sets `limits` explicitly before passing to the Client), and any caller that needs different limits can still pass `limits=...` via kwargs.
Choosing `(5, 20)` rather than `(1, 10)`: a non-supervisor Client (e.g. instantiated directly by tests, CLI, or future callers) may handle higher per-instance concurrency than a single task subprocess, so the default sits between the supervisor's tight cap and httpx's untenably-high default.
## Test plan
- [x] `test_default_pool_limits_are_bounded` — instantiates a real (non-mock-transport) Client and asserts `pool._max_connections == 20` and `pool._max_keepalive_connections == 5`.
- [x] `test_pool_limits_can_be_overridden` — `limits=httpx.Limits(...)` kwarg overrides the default.
- [x] `prek run ruff` clean.
- [x] `prek run mypy-task-sdk` clean.
- [x] Full `test_client.py` suite: 117 passed.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 4.7)
Generated-by: Claude Code (Opus 4.7) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Cap default httpx connection pool on the task-sdk Client [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #67506:
URL: https://github.com/apache/airflow/pull/67506#issuecomment-4627304154
You're right — closing this. Traced it through: the only real-network instantiation is `_ensure_client` in the supervisor, which already sets `limits=httpx.Limits(max_connections=10)` explicitly, so the `setdefault("limits", …)` here never fires on that path. The other two call sites (`dag_processing/manager.py`, `triggerer_job_runner.py`) omit `limits` but use in-process dry-run transports, where pool limits are moot. So the cap is effectively a no-op and the comment's "task subprocess / both code paths" framing was wrong. Thanks for catching it.
---
Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Cap default httpx connection pool on the task-sdk Client [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #67506: Cap default httpx connection pool on the task-sdk Client
URL: https://github.com/apache/airflow/pull/67506
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Cap default httpx connection pool on the task-sdk Client [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #67506:
URL: https://github.com/apache/airflow/pull/67506#discussion_r3353493492
##########
task-sdk/src/airflow/sdk/api/client.py:
##########
@@ -1136,6 +1136,13 @@ def __init__(self, *, base_url: str | None, dry_run: bool = False, token: str, *
# Set timeout if not explicitly provided
kwargs.setdefault("timeout", API_TIMEOUT)
+ # Cap the httpx connection pool. Without an explicit value httpx defaults to 100 max
+ # connections, which is far higher than a single task subprocess ever needs (the
+ # supervisor sets ``max_connections=10`` in ``_ensure_client``). A bounded default
Review Comment:
This code is only run in the supervisor, so this comment makes no sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org