You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Taragolis (via GitHub)" <gi...@apache.org> on 2024/05/24 09:18:29 UTC
Re: [PR] Change `httpx` to `requests` in `file_task_handler` [airflow]
Taragolis commented on PR #39799:
URL: https://github.com/apache/airflow/pull/39799#issuecomment-2129045571
I guess the main prom that there is no standardisation for [`no_proxy`](https://about.gitlab.com/blog/2021/01/27/we-need-to-talk-no-proxy/) and different libraries / os could work differently
Most of the libraries use stdlib implementation from [Lib/urllib/request.py](https://github.com/python/cpython/tree/3.12/Lib/urllib/request.py)
Request use own implementation which supports CIDR blocks, but only for ipv4:
- https://github.com/psf/requests/issues/6313
```python
import os
os.environ["no_proxy"] = "localhost,127.0.0.1,10.0.0.0/8,172.16.1.*,1:2:3::/64"
os.environ["NO_PROXY"] = "localhost,127.0.0.1,192.168.1.0/16"
# stdlib implementation
from urllib.request import proxy_bypass_environment
# requests implementation
from requests.sessions import should_bypass_proxies
hosts = ("localhost", "127.0.0.1", "127.0.0.2", "10.0.0.0", "192.168.1.32", "172.16.1.2", "[1:2:3::4]")
for host in hosts:
print(f" {host} ".center(80, '='))
print(f"stdlib Proxy Bypass Environment: {proxy_bypass_environment(host, None)}")
print(f"requests Proxy Bypass Environment: {should_bypass_proxies(f"http://{host}", None)}")
```
```console
================================== localhost ===================================
stdlib Proxy Bypass Environment: True
requests Proxy Bypass Environment: True
================================== 127.0.0.1 ===================================
stdlib Proxy Bypass Environment: True
requests Proxy Bypass Environment: True
================================== 127.0.0.2 ===================================
stdlib Proxy Bypass Environment: False
requests Proxy Bypass Environment: False
=================================== 10.0.0.0 ===================================
stdlib Proxy Bypass Environment: False
requests Proxy Bypass Environment: True
================================= 192.168.1.32 =================================
stdlib Proxy Bypass Environment: False
requests Proxy Bypass Environment: False
================================== 172.16.1.2 ==================================
stdlib Proxy Bypass Environment: False
requests Proxy Bypass Environment: False
================================== [1:2:3::4] ==================================
stdlib Proxy Bypass Environment: False
requests Proxy Bypass Environment: False
```
I don't know what is supposed to be happen in case if we change library in the future, e.g. from `requests` to `urllib3`, or use `aiohttp` in case if we turn internal code to async implementation (`asyncio` / `anyio` / `trio`)
So probably it is fine for replace `httpx` by `requests`, and I'm pretty sure that in past this was called by requests before we temporary remove usage of requests due to licence issue: https://github.com/apache/airflow/pull/15781. However there is **should not be a commitment that this behaviour remaining the same** in the future, because this part of the code usually can't be changed without monkey patching third-party library / stdlib
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org