You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "1fanwang (via GitHub)" <gi...@apache.org> on 2026/09/29 23:27:15 UTC
[PR] Allow partial pool updates with update_mask [airflow]
1fanwang opened a new pull request, #73931:
URL: https://github.com/apache/airflow/pull/73931
A client can't change just the slots of a pool. `PATCH /api/v2/pools/{pool_name}?update_mask=slots` with `{"slots": 8}` returns 422 because the pool name and `include_deferred` are still required. A bulk update with the same mask fails too, and only `default_pool` honours the mask. So clients resend `include_deferred` every time, and `airflowctl pools update --pool etl --slots 16` sends false, which turns deferred slots off on a pool that had them on.
With this change, a PATCH or bulk update with a mask needs only the masked fields and keeps the rest. Masked fields are still validated (`{"slots": null}` is still a 422), and a PATCH without a mask still needs the full body.
The service validated the body against the full pool schema before applying the mask. Now it validates only the fields it will write and takes the other required ones from the stored pool.
closes: https://github.com/apache/airflow/issues/73768
related: https://github.com/apache/airflow/pull/71220 edits the same validation block. They are compatible; the second to merge needs a rebase there.
### Testing Done
The reproducer below drives a real API server (SQLite, simple auth manager) run with `airflow` from `uv sync --project airflow-core`, once at main 18fe1c6 and once at this branch.
Before, main at 18fe1c6:
```
$ bash pool-mask-e2e.sh .venv/bin/airflow
$ airflow version # the server
3.4.0
$ curl -X POST http://127.0.0.1:64263/api/v2/pools -d '{"name": "etl", "slots": 4, "description": "nightly loads", "include_deferred": true}'
{"name":"etl","slots":4,"description":"nightly loads","include_deferred":true,"occupied_slots":0,"running_slots":0,"queued_slots":0,"scheduled_slots":0,"open_slots":4,"deferred_slots":0,"team_name":null}
HTTP 201
$ curl -X PATCH 'http://127.0.0.1:64263/api/v2/pools/etl?update_mask=slots' -d '{"slots": 8}'
{"detail":[{"type":"missing","loc":["pool"],"msg":"Field required","input":{"slots":8},"url":"https://errors.pydantic.dev/2.13/v/missing"},{"type":"missing","loc":["include_deferred"],"msg":"Field required","input":{"slots":8},"url":"https://errors.pydantic.dev/2.13/v/missing"}]}
HTTP 422
$ curl -X PATCH http://127.0.0.1:64263/api/v2/pools -d '{"actions": [{"action": "update", "entities": [{"name": "etl", "slots": 12}], "update_mask": ["slots"]}]}'
{"detail":[{"type":"missing","loc":["include_deferred"],"msg":"Field required","input":{"slots":12,"pool":"etl"},"url":"https://errors.pydantic.dev/2.13/v/missing"}]}
HTTP 422
$ curl http://127.0.0.1:64263/api/v2/pools/etl
{"name":"etl","slots":4,"description":"nightly loads","include_deferred":true,"occupied_slots":0,"running_slots":0,"queued_slots":0,"scheduled_slots":0,"open_slots":4,"deferred_slots":0,"team_name":null}
HTTP 200
$ curl -X PATCH 'http://127.0.0.1:64263/api/v2/pools/etl?update_mask=slots' -d '{"slots": null}' # a masked field is still validated
{"detail":[{"type":"missing","loc":["pool"],"msg":"Field required","input":{"slots":null},"url":"https://errors.pydantic.dev/2.13/v/missing"},{"type":"int_type","loc":["slots"],"msg":"Input should be a valid integer","input":null,"url":"https://errors.pydantic.dev/2.13/v/int_type"},{"type":"missing","loc":["include_deferred"],"msg":"Field required","input":{"slots":null},"url":"https://errors.pydantic.dev/2.13/v/missing"}]}
HTTP 422
$ curl -X PATCH http://127.0.0.1:64263/api/v2/pools/etl -d '{"slots": 8}' # without update_mask the full body is still required
{"detail":[{"type":"missing","loc":["pool"],"msg":"Field required","input":{"slots":8},"url":"https://errors.pydantic.dev/2.13/v/missing"},{"type":"missing","loc":["include_deferred"],"msg":"Field required","input":{"slots":8},"url":"https://errors.pydantic.dev/2.13/v/missing"}]}
HTTP 422
```
After, this branch:
```
$ bash pool-mask-e2e.sh .venv/bin/airflow
$ airflow version # the server
3.4.0
$ curl -X POST http://127.0.0.1:64318/api/v2/pools -d '{"name": "etl", "slots": 4, "description": "nightly loads", "include_deferred": true}'
{"name":"etl","slots":4,"description":"nightly loads","include_deferred":true,"occupied_slots":0,"running_slots":0,"queued_slots":0,"scheduled_slots":0,"open_slots":4,"deferred_slots":0,"team_name":null}
HTTP 201
$ curl -X PATCH 'http://127.0.0.1:64318/api/v2/pools/etl?update_mask=slots' -d '{"slots": 8}'
{"name":"etl","slots":8,"description":"nightly loads","include_deferred":true,"occupied_slots":0,"running_slots":0,"queued_slots":0,"scheduled_slots":0,"open_slots":8,"deferred_slots":0,"team_name":null}
HTTP 200
$ curl -X PATCH http://127.0.0.1:64318/api/v2/pools -d '{"actions": [{"action": "update", "entities": [{"name": "etl", "slots": 12}], "update_mask": ["slots"]}]}'
{"create":null,"update":{"success":["etl"],"errors":[]},"delete":null}
HTTP 200
$ curl http://127.0.0.1:64318/api/v2/pools/etl
{"name":"etl","slots":12,"description":"nightly loads","include_deferred":true,"occupied_slots":0,"running_slots":0,"queued_slots":0,"scheduled_slots":0,"open_slots":12,"deferred_slots":0,"team_name":null}
HTTP 200
$ curl -X PATCH 'http://127.0.0.1:64318/api/v2/pools/etl?update_mask=slots' -d '{"slots": null}' # a masked field is still validated
{"detail":[{"type":"int_type","loc":["slots"],"msg":"Input should be a valid integer","input":null,"url":"https://errors.pydantic.dev/2.13/v/int_type"}]}
HTTP 422
$ curl -X PATCH http://127.0.0.1:64318/api/v2/pools/etl -d '{"slots": 8}' # without update_mask the full body is still required
{"detail":[{"type":"missing","loc":["pool"],"msg":"Field required","input":{"slots":8},"url":"https://errors.pydantic.dev/2.13/v/missing"},{"type":"missing","loc":["include_deferred"],"msg":"Field required","input":{"slots":8},"url":"https://errors.pydantic.dev/2.13/v/missing"}]}
HTTP 422
```
<details>
<summary>Reproducer source: pool-mask-e2e.sh</summary>
```bash
#!/usr/bin/env bash
# Partial pool PATCH requests with and without update_mask against a real Airflow API server (SQLite, simple auth manager).
# Usage: pool-mask-e2e.sh <airflow executable for the server>
set -u
AIRFLOW=${1:?usage: pool-mask-e2e.sh <airflow>}
PORT=$(python3 -c 'import socket; s = socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1])')
URL="http://127.0.0.1:$PORT"
SERVER_HOME="$(mktemp -d)"
SERVER_ENV=(
AIRFLOW_HOME="$SERVER_HOME"
AIRFLOW__CORE__LOAD_EXAMPLES=False
AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_USERS=admin:admin
)
say() { printf '\n$ %s\n' "$1"; }
env "${SERVER_ENV[@]}" "$AIRFLOW" db migrate > "$SERVER_HOME/setup.log" 2>&1 || { cat "$SERVER_HOME/setup.log"; exit 1; }
env "${SERVER_ENV[@]}" "$AIRFLOW" api-server --port "$PORT" --workers 1 > "$SERVER_HOME/api-server.log" 2>&1 &
SERVER_PID=$!
trap 'kill "$SERVER_PID" 2>/dev/null; wait "$SERVER_PID" 2>/dev/null; rm -rf "$SERVER_HOME"' EXIT
PASSWORDS="$SERVER_HOME/simple_auth_manager_passwords.json.generated"
for _ in $(seq 90); do
curl -sf "$URL/api/v2/monitor/health" > /dev/null && [ -s "$PASSWORDS" ] && break
sleep 1
done
PASSWORD=$(python3 -c 'import json, sys; print(json.load(open(sys.argv[1]))["admin"])' "$PASSWORDS")
TOKEN=$(curl -s -X POST "$URL/auth/token" -H 'Content-Type: application/json' \
-d "{\"username\": \"admin\", \"password\": \"$PASSWORD\"}" | python3 -c 'import json, sys; print(json.load(sys.stdin)["access_token"])')
api() { curl -s -w '\nHTTP %{http_code}\n' -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' "$@"; }
say 'airflow version # the server'
env "${SERVER_ENV[@]}" "$AIRFLOW" version 2> /dev/null
CREATE='{"name": "etl", "slots": 4, "description": "nightly loads", "include_deferred": true}'
say "curl -X POST $URL/api/v2/pools -d '$CREATE'"
api -X POST "$URL/api/v2/pools" -d "$CREATE"
say "curl -X PATCH '$URL/api/v2/pools/etl?update_mask=slots' -d '{\"slots\": 8}'"
api -X PATCH "$URL/api/v2/pools/etl?update_mask=slots" -d '{"slots": 8}'
BULK='{"actions": [{"action": "update", "entities": [{"name": "etl", "slots": 12}], "update_mask": ["slots"]}]}'
say "curl -X PATCH $URL/api/v2/pools -d '$BULK'"
api -X PATCH "$URL/api/v2/pools" -d "$BULK"
say "curl $URL/api/v2/pools/etl"
api "$URL/api/v2/pools/etl"
say "curl -X PATCH '$URL/api/v2/pools/etl?update_mask=slots' -d '{\"slots\": null}' # a masked field is still validated"
api -X PATCH "$URL/api/v2/pools/etl?update_mask=slots" -d '{"slots": null}'
say "curl -X PATCH $URL/api/v2/pools/etl -d '{\"slots\": 8}' # without update_mask the full body is still required"
api -X PATCH "$URL/api/v2/pools/etl" -d '{"slots": 8}'
```
</details>
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: GitHub Copilot CLI (Claude Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org