You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kaxil (via GitHub)" <gi...@apache.org> on 2026/09/09 23:59:00 UTC
[PR] Fix PydanticAI vendor connections not resolving from secrets backends [airflow]
kaxil opened a new pull request, #72853:
URL: https://github.com/apache/airflow/pull/72853
## Summary
The three PydanticAI vendor hooks declared their `conn_type` with hyphens, which made them unreachable from every secrets backend. A hook is registered under the literal `connection-type` string from `provider.yaml`, but [`Connection.from_uri`](https://github.com/apache/airflow/blob/251cd91bb2/airflow-core/src/airflow/models/connection.py#L235) and `Connection.from_json` both run [`_normalize_conn_type`](https://github.com/apache/airflow/blob/251cd91bb2/airflow-core/src/airflow/models/connection.py#L228-L233), which rewrites `-` to `_` before the lookup. The key the hook is [registered under](https://github.com/apache/airflow/blob/251cd91bb2/airflow-core/src/airflow/providers_manager.py#L679-L706) could therefore never be produced by either parse path, and any task using one of these connections failed with:
```
AirflowException: Unknown hook type "pydanticai_azure"
File ".../airflow/sdk/definitions/connection.py", line 223 in get_hook
```
A connection read straight out of the metadata DB kept its hyphen and resolved, which is why this went unnoticed. Even that path was only reliable with the secret cache off: [`get_connection_from_secrets`](https://github.com/apache/airflow/blob/251cd91bb2/airflow-core/src/airflow/models/connection.py#L521) caches `conn.get_uri()` and rebuilds the connection from that URI on the next read, so with `secrets.use_cache = True` a DB-stored hyphenated connection worked on first use and failed on every subsequent one.
Closes #72316. Supersedes #72341, which identified the same root cause.
## The blast radius is wider than the report
The report describes URI-valued backends such as HashiCorp Vault. `from_json` normalizes identically, and so does the `AIRFLOW_CONN_*` environment-variable backend, which is enabled by default and is how most deployments define connections. All six combinations below failed before this change and resolve after it:
Both stored shapes are exercised because both occur in the wild. JSON carries the canonical underscore `conn_type`; a URI's scheme is hyphenated because RFC 3986 forbids `_` in a scheme, so `Connection.get_uri()` writes hyphens and `_normalize_conn_type` converts back on read. Hyphenated URIs already sitting in a secrets backend keep working.
## Why rename rather than also register the hyphenated form
Three ways to close this, and why the rename wins:
- **Add the hyphenated names to `_normalize_conn_type`'s alias table**, where `postgresql` → `postgres` lives. That puts provider-specific names in airflow-core, for one provider's benefit.
- **Register a second hook class per vendor under the hyphenated key.** This does not fix the reported path at all: a hyphenated URI or JSON value already normalizes onto the underscore key, so the extra registration only serves connections sitting hyphenated in the metadata DB. It also adds three unlabelled entries to the connection-type dropdown, because a `connection-types` entry with no `hook-name` falls back to the raw slug.
- **Rename**, which matches every other multi-word hook in the tree (`azure_data_factory`, `google_cloud_platform`, `spark_connect`, `hive_cli`) and leaves exactly one canonical name per type.
One entry per type after the rename, each with its own label:
## Migration
`default_conn_name` values were already underscored (`pydanticai_azure_default`) and are unchanged, so only `conn_type` moves. A connection already stored with the hyphenated type needs its type corrected once:
```bash
airflow connections delete my_azure_conn
airflow connections add my_azure_conn \
--conn-type pydanticai_azure \
--conn-host https://my-resource.openai.azure.com \
--conn-password '<api-key>' \
--conn-extra '{"model": "azure:gpt-4o", "api_version": "2024-07-01-preview"}'
```
In the UI, re-pick the type on the existing connection. No compatibility shim is included: these three types are recent, the provider is pre-1.0, and a permanent second registration per vendor is not worth the dropdown noise it costs every user.
## Tradeoff worth stating
Underscores bring back a warning whenever a connection is serialized to a URI:
```
Connection schemes (type: pydanticai_azure) shall not contain '_' according to RFC3986.
```
That warning is exactly why `pydantic_ai` became `pydanticai` in #62817. It is a log line only, the hyphenated URI it emits is valid and still round-trips, and `azure_data_factory` and `google_cloud_platform` have carried the same warning for years. Correct hook resolution is worth more than a clean log line.
## Guard against a repeat
Nothing stopped a hyphen getting in: `connection-type` was declared as a bare `{"type": "string"}` in both provider schemas. This adds `"pattern": "^[a-z0-9_]+$"` there, so the same mistake now fails schema validation at authoring time, for any provider, rather than at a user's first task run. All 141 `connection-type` values across the provider tree already satisfy it, so it lands green.
Two smaller guards alongside it. `TestConnTypeResolution` asserts that every `connection-type` this provider declares survives a URI round-trip, parametrized so it names every offender rather than aborting at the first. And the `connection-types` example in `contributing-docs/23_provider_hook_migration_to_yaml.rst` used the hyphenated form as its illustration, which is what a new provider author copies, so it is corrected too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix PydanticAI vendor connections not resolving from secrets backends [airflow]
Posted by "pankajastro (via GitHub)" <gi...@apache.org>.
pankajastro commented on code in PR #72853:
URL: https://github.com/apache/airflow/pull/72853#discussion_r3982139114
##########
airflow-core/src/airflow/provider.yaml.schema.json:
##########
@@ -428,8 +428,9 @@
"type": "object",
"properties": {
"connection-type": {
- "description": "Type of connection defined by the provider",
- "type": "string"
+ "description": "Type of connection defined by the provider. Lowercase letters, digits and '_' only, starting with a letter. The hook is registered under this exact string, while Connection.get_uri() encodes '_' as '-' and from_uri/from_json decode '-' back to '_'. A '-' here is therefore indistinguishable from an encoded '_' and leaves the hook unresolvable for any connection stored as a URI or JSON; uppercase is lost to get_uri()'s lowercasing; and a leading digit is not a valid URI scheme under RFC 3986.",
+ "type": "string",
+ "pattern": "^[a-z][a-z0-9_]*$"
Review Comment:
jsonschema's Python regex backend uses re.search, not re.fullmatch — and $ matches just before a trailing \n, not only end-of-string. Does ^[a-z][a-z0-9_]*$ actually reject a connection-type value with an embedded trailing newline (e.g. from a YAML block scalar), or would that slip through check-provider-yaml-valid the same way the hyphenated values did?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix PydanticAI vendor connections not resolving from secrets backends [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #72853:
URL: https://github.com/apache/airflow/pull/72853#discussion_r3983078459
##########
airflow-core/src/airflow/provider.yaml.schema.json:
##########
@@ -428,8 +428,9 @@
"type": "object",
"properties": {
"connection-type": {
- "description": "Type of connection defined by the provider",
- "type": "string"
+ "description": "Type of connection defined by the provider. Lowercase letters, digits and '_' only, starting with a letter. The hook is registered under this exact string, while Connection.get_uri() encodes '_' as '-' and from_uri/from_json decode '-' back to '_'. A '-' here is therefore indistinguishable from an encoded '_' and leaves the hook unresolvable for any connection stored as a URI or JSON; uppercase is lost to get_uri()'s lowercasing; and a leading digit is not a valid URI scheme under RFC 3986.",
+ "type": "string",
+ "pattern": "^[a-z][a-z0-9_]*$"
Review Comment:
addressed in https://github.com/apache/airflow/pull/72853/changes/5d5e1f51f10c401fe3ed4d9909368fe7032225f5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix PydanticAI vendor connections not resolving from secrets backends [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil merged PR #72853:
URL: https://github.com/apache/airflow/pull/72853
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org