You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "nailo2c (via GitHub)" <gi...@apache.org> on 2025/06/22 22:57:12 UTC
[PR] Fix HttpOperator(deferrable=True) crash when connection has login / password [airflow]
nailo2c opened a new pull request, #52050:
URL: https://github.com/apache/airflow/pull/52050
Closes: #46863
# Why
If an HTTP connection contains login/password and the user doesn't set `auth_type`, the sync path works (the hook auto-defaults to HTTPBasicAuth), but the deferrable path crashes:
https://github.com/apache/airflow/blob/a8a9fc5ead28ab4902a328a6911fe2c95e737776/providers/http/src/airflow/providers/http/operators/http.py#L220-L224
passes `None` to `auth_type`, store it in the `trigger` table.
```markdown
id | classpath | kwargs | created_date | triggerer_id
----+--------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------+--------------
3 | airflow.providers.http.triggers.http.HttpTrigger | {"__var": {"http_conn_id": "http_default", "method": "GET", "auth_type": null, "endpoint": "get", "headers": {"__var": {}, "__type": "dict"}, "data": {"__var": {}, "__type": "dict"}, "extra_options": {"__var": {}, "__type": "dict"}}, "__type": "dict"} | 2025-06-18 04:01:51.49263+00 | 3
(1 row)
```
after deserialization, `None` is passed here, causing `'NoneType' object is not callable` error.
https://github.com/apache/airflow/blob/a8a9fc5ead28ab4902a328a6911fe2c95e737776/providers/http/src/airflow/providers/http/hooks/http.py#L439-L440
# How
1. Add `_resolve_auth_type()` to **HttpOperator**;
it auto-picks `aiohttp.BasicAuth` (when the conn has login/password) before deferral.
2. Add `serialize_auth_type()`; always store the fully-qualified class path or `None`.
3. Add `deserialize_auth_type()`;
`HttpTrigger.__init__()` uses it to turn the stored string back **into the class object** at runtime.
4. Serialize `auth_type` when `HttpOperator.execute_async()` and deserialises it when `HttpTrigger.__init__()`,
so internal logic remains unchanged.
# What
Test DAG
```python
from airflow import DAG
from airflow.providers.http.operators.http import HttpOperator
import pendulum
with DAG(
dag_id="http_deferrable_bug_demo",
start_date=pendulum.datetime(2025, 1, 1, tz="UTC"),
schedule=None,
catchup=False,
) as dag:
HttpOperator(
task_id="call_httpbin",
http_conn_id="http_default",
endpoint="get",
method="GET",
deferrable=True,
)
```
Test connection
```console
airflow connections add 'http_default' \
--conn-type http \
--conn-host 'https://httpbin.org' \
--conn-login 'user' \
--conn-password 'pass'
```
before
<img width="1816" alt="截圖 2025-06-22 auth_type is None" src="https://github.com/user-attachments/assets/7a5eb3d3-0c51-47c3-9570-0728ddc33aef" />
after
<img width="1817" alt="截圖 2025-06-22 auth_type is not None" src="https://github.com/user-attachments/assets/648171a4-b281-44cd-87e5-98cab99f6d26" />
the `auth_type` in the `trigger` table
```markdown
id | classpath | kwargs | created_date | triggerer_id
----+--------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------+--------------
1 | airflow.providers.http.triggers.http.HttpTrigger | {"__var": {"http_conn_id": "http_default", "method": "GET", "auth_type": "aiohttp.helpers.BasicAuth", "endpoint": "get", "headers": {"__var": {}, "__type": "dict"}, "data": {"__var": {}, "__type": "dict"}, "extra_options": {"__var": {}, "__type": "dict"}}, "__type": "dict"} | 2025-06-19 22:52:32.606647+00 | 2
(1 row)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix HttpOperator(deferrable=True) crash when connection has login / password [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #52050:
URL: https://github.com/apache/airflow/pull/52050#issuecomment-2994903539
Great catch and good implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix HttpOperator(deferrable=True) crash when connection has login / password [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #52050:
URL: https://github.com/apache/airflow/pull/52050
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org