You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "alex-brunshtein (via GitHub)" <gi...@apache.org> on 2025/12/03 13:52:51 UTC

[I] Creating a virtual environment via uv venv fail without direct internet access [airflow]

alex-brunshtein opened a new issue, #58999:
URL: https://github.com/apache/airflow/issues/58999

   ### Apache Airflow Provider(s)
   
   standard
   
   ### Versions of Apache Airflow Providers
   
   [apache-airflow-providers-standard:1.9.1](https://airflow.apache.org/docs/apache-airflow-providers-standard/1.9.1/)
   
   ### Apache Airflow version
   
   3.1.3
   
   ### Operating System
   
   Debian GNU/Linux 12 (bookworm)
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   k8s
   
   ### What happened
   
   Creating a virtual environment via uv venv in the PythonVirtualenvOperator fails without direct internet access. Access to https://pypi.org/simple/pip/ provided through Artifactory in transparent proxy mode. 
   A corresponding package-index connection "remote-pypi" has been configured. 
   When using uv, this connection is only applied during package installation, but not during virtual environment creation:
   
   ```
       python_virtualenv_operator = PythonVirtualenvOperator(
           task_id="python_virtualenv_operator",
           python_callable=python_virtualenv_code,
           requirements=["certifi"],
           index_urls_from_connection_ids=["remote-pypi"],
       )
   
   [2025-12-03 15:29:25] INFO - DAG bundles loaded: dags-folder, example_dags source=airflow.dag_processing.bundles.manager.DagBundlesManager loc=manager.py:179
   [2025-12-03 15:29:25] INFO - Filling up the DagBag from /opt/airflow/dags/repo/dags/repo1/airflow-3/python_operators.py source=airflow.models.dagbag.DagBag loc=dagbag.py:593
   [2025-12-03 15:29:25] INFO - Task instance is in running state source=task.stdout
   [2025-12-03 15:29:25] INFO -  Previous state of the Task instance: TaskInstanceState.QUEUED source=task.stdout
   [2025-12-03 15:29:25] INFO - Current task name:python_virtualenv_operator source=task.stdout
   [2025-12-03 15:29:25] INFO - Dag name:af3_python_operators source=task.stdout
   [2025-12-03 15:29:25] WARNING - Connection schemes (type: package_index) shall not contain '_' according to RFC3986. source=airflow.sdk.definitions.connection loc=connection.py:130
   [2025-12-03 15:29:25] INFO - Executing cmd: uv venv --allow-existing --seed --python python --system-site-packages /tmp/venv2c51r6_g source=airflow.utils.process_utils loc=process_utils.py:188
   [2025-12-03 15:29:25] INFO - Output: source=airflow.utils.process_utils loc=process_utils.py:192
   [2025-12-03 15:29:26] INFO - Using CPython 3.12.7 interpreter at: /usr/local/bin/python source=airflow.utils.process_utils loc=process_utils.py:196
   [2025-12-03 15:29:26] INFO - Creating virtual environment with seed packages at: /tmp/venv2c51r6_g source=airflow.utils.process_utils loc=process_utils.py:196
   [2025-12-03 15:31:31] INFO - uv::venv::seed source=airflow.utils.process_utils loc=process_utils.py:196
   [2025-12-03 15:31:31] INFO -  source=airflow.utils.process_utils loc=process_utils.py:196
   [2025-12-03 15:31:31] INFO -   × Failed to install seed packages source=airflow.utils.process_utils loc=process_utils.py:196
   [2025-12-03 15:31:31] INFO -   ├─▶ No solution found when resolving: `pip` source=airflow.utils.process_utils loc=process_utils.py:196
   [2025-12-03 15:31:31] INFO -   ├─▶ Failed to fetch: `https://pypi.org/simple/pip/` source=airflow.utils.process_utils loc=process_utils.py:196
   [2025-12-03 15:31:31] INFO -   ├─▶ Request failed after 3 retries source=airflow.utils.process_utils loc=process_utils.py:196
   [2025-12-03 15:31:31] INFO -   ├─▶ error sending request for url (https://pypi.org/simple/pip/) source=airflow.utils.process_utils loc=process_utils.py:196
   [2025-12-03 15:31:31] INFO -   ╰─▶ operation timed out source=airflow.utils.process_utils loc=process_utils.py:196
   [2025-12-03 15:31:31] ERROR - Task failed with exception source=task loc=task_runner.py:1002
   CalledProcessError: Command '['uv', 'venv', '--allow-existing', '--seed', '--python', 'python', '--system-site-packages', '/tmp/venv2c51r6_g']' returned non-zero exit status 1.
   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 928 in run
   
   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1315 in _execute_task
   
   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 416 in wrapper
   
   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 488 in execute
   
   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 416 in wrapper
   
   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 213 in execute
   
   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 892 in execute_callable
   
   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 770 in _prepare_venv
   
   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/utils/python_virtualenv.py", line 172 in prepare_virtualenv
   
   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/process_utils.py", line 177 in execute_in_subprocess
   
   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/process_utils.py", line 200 in execute_in_subprocess_with_kwargs
   
   [2025-12-03 15:31:31] INFO - Task instance in failure state source=task.stdout
   
   ```
   
   
   ### What you think should happen instead
   
   Creating venv via uv not use UV_INDEX_URL and use --seed flag:
   `cmd = ["uv", "venv", "--allow-existing", "--seed", "--python", python_bin]`
   
   ### How to reproduce
   
   Use uv for package installation
   Block direct internet access. 
   Use only package-index connections to install common packages 
   
   ### Anything else
   
   It's worked if adding the environment variable UV_INDEX_URL for venv creation (similar to how it's handled during package installation).
   Needs change [python_virtualenv.py](https://github.com/apache/airflow/blob/main/providers/standard/src/airflow/providers/standard/utils/python_virtualenv.py#L205:~:text=Line-,205,-options)
   For example, change
   `_execute_in_subprocess(venv_cmd)`
   to
   `_execute_in_subprocess(venv_cmd, env={**os.environ, **_index_urls_to_uv_env_vars(index_urls)})`
   as for `pip_cmd`
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Creating a virtual environment via uv venv fail without direct internet access [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #58999:
URL: https://github.com/apache/airflow/issues/58999#issuecomment-3606965197

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Creating a virtual environment via uv venv fail without direct internet access [airflow]

Posted by "henry3260 (via GitHub)" <gi...@apache.org>.
henry3260 commented on issue #58999:
URL: https://github.com/apache/airflow/issues/58999#issuecomment-3609844579

   Can I work on this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Creating a virtual environment via uv venv fail without direct internet access [airflow]

Posted by "Arunodoy18 (via GitHub)" <gi...@apache.org>.
Arunodoy18 commented on issue #58999:
URL: https://github.com/apache/airflow/issues/58999#issuecomment-3610682787

   Hi, i think i can solve this issue , I would love to contribute on this issue , .take-issue.
   Thank you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Creating a virtual environment via uv venv fail without direct internet access [airflow]

Posted by "alex-brunshtein (via GitHub)" <gi...@apache.org>.
alex-brunshtein commented on issue #58999:
URL: https://github.com/apache/airflow/issues/58999#issuecomment-3609904757

   Yes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Creating a virtual environment via uv venv fail without direct internet access [airflow]

Posted by "alex-brunshtein (via GitHub)" <gi...@apache.org>.
alex-brunshtein commented on issue #58999:
URL: https://github.com/apache/airflow/issues/58999#issuecomment-3670422754

   uv venv works great! 
   Many thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Creating a virtual environment via uv venv fail without direct internet access [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #58999: Creating a virtual environment via uv venv fail without direct internet access
URL: https://github.com/apache/airflow/issues/58999


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org