You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Taragolis (via GitHub)" <gi...@apache.org> on 2024/05/24 09:18:29 UTC

Re: [PR] Change `httpx` to `requests` in `file_task_handler` [airflow]

Taragolis commented on PR #39799:
URL: https://github.com/apache/airflow/pull/39799#issuecomment-2129045571

   I guess the main prom that there is no standardisation for [`no_proxy`](https://about.gitlab.com/blog/2021/01/27/we-need-to-talk-no-proxy/) and different libraries / os could work differently
   
   Most of the libraries use stdlib implementation from [Lib/urllib/request.py](https://github.com/python/cpython/tree/3.12/Lib/urllib/request.py)
   
   Request use own implementation which supports CIDR blocks, but only for ipv4:
   - https://github.com/psf/requests/issues/6313
   
   ```python
   import os
   
   os.environ["no_proxy"] = "localhost,127.0.0.1,10.0.0.0/8,172.16.1.*,1:2:3::/64"
   os.environ["NO_PROXY"] = "localhost,127.0.0.1,192.168.1.0/16"
   
   # stdlib implementation
   from urllib.request import proxy_bypass_environment
   # requests implementation
   from requests.sessions import should_bypass_proxies
   
   hosts = ("localhost", "127.0.0.1", "127.0.0.2", "10.0.0.0", "192.168.1.32", "172.16.1.2", "[1:2:3::4]")
   for host in hosts:
       print(f" {host} ".center(80, '='))
       print(f"stdlib Proxy Bypass Environment: {proxy_bypass_environment(host, None)}")
       print(f"requests Proxy Bypass Environment: {should_bypass_proxies(f"http://{host}", None)}")
   ```
   
   ```console
   ================================== localhost ===================================
   stdlib Proxy Bypass Environment: True
   requests Proxy Bypass Environment: True
   ================================== 127.0.0.1 ===================================
   stdlib Proxy Bypass Environment: True
   requests Proxy Bypass Environment: True
   ================================== 127.0.0.2 ===================================
   stdlib Proxy Bypass Environment: False
   requests Proxy Bypass Environment: False
   =================================== 10.0.0.0 ===================================
   stdlib Proxy Bypass Environment: False
   requests Proxy Bypass Environment: True
   ================================= 192.168.1.32 =================================
   stdlib Proxy Bypass Environment: False
   requests Proxy Bypass Environment: False
   ================================== 172.16.1.2 ==================================
   stdlib Proxy Bypass Environment: False
   requests Proxy Bypass Environment: False
   ================================== [1:2:3::4] ==================================
   stdlib Proxy Bypass Environment: False
   requests Proxy Bypass Environment: False
   ```
   
   I don't know what is supposed to be happen in case if we change library in the future, e.g. from `requests` to `urllib3`, or use `aiohttp` in case if we turn internal code to async implementation (`asyncio` / `anyio` / `trio`)
   
   So probably it is fine for replace `httpx` by `requests`, and I'm pretty sure that in past this was called by requests before we temporary remove usage of requests due to licence issue: https://github.com/apache/airflow/pull/15781. However there is **should not be a commitment that this behaviour remaining the same** in the future, because this part of the code usually can't be changed without monkey patching third-party library / stdlib


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org