You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Pappol (via GitHub)" <gi...@apache.org> on 2025/05/23 10:19:25 UTC
[I] Add uv venv implementation [airflow]
Pappol opened a new issue, #50999:
URL: https://github.com/apache/airflow/issues/50999
### Description
uv is a lot faster than pip, it would be nice to have a uv implementation of virtual env tasks
### Use case/motivation
Arflow dags are very slow in the case of venv tasks, this could notably speed up the process
### Related issues
There is no control over how a venv is created at such level
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add uv venv implementation [airflow]
Posted by "tadeha (via GitHub)" <gi...@apache.org>.
tadeha commented on issue #50999:
URL: https://github.com/apache/airflow/issues/50999#issuecomment-3023877359
Hi, any updates on it? We also think this would be a great addition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add uv venv implementation [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #50999:
URL: https://github.com/apache/airflow/issues/50999#issuecomment-2903973090
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add uv venv implementation [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #50999: Add uv venv implementation
URL: https://github.com/apache/airflow/issues/50999
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add uv venv implementation [airflow]
Posted by "tadeha (via GitHub)" <gi...@apache.org>.
tadeha commented on issue #50999:
URL: https://github.com/apache/airflow/issues/50999#issuecomment-3027775270
@potiuk Thank you! We're using the Airflow Helm Chart version 1.16.0 with Airflow 2.10.5. I've set the environment variable AIRFLOW__STANDARD__VENV_INSTALL_METHOD=uv, but in DAG runs that use virtual environments, I still see pip being logged. Should I be seeing something different?
I’ve confirmed that the environment variable is set correctly and that uv is installed.
DAG example:
```
import pendulum
from airflow.models.dag import DAG
from airflow.operators.python import PythonVirtualenvOperator
def my_virtualenv_callable():
import pandas as pd
print(f"Pandas version: {pd.__version__}")
data = {'col1': [1, 2], 'col2': [3, 4]}
df = pd.DataFrame(data)
print("DataFrame created successfully:")
print(df)
# You can also return values, but they must be pickleable
return "Virtual environment task completed!"
with DAG(
dag_id="test_venv_dag",
start_date=pendulum.datetime(2025, 7, 2, tz="UTC"),
schedule=None,
catchup=False,
tags=["virtualenv"],
) as dag:
run_in_venv_task = PythonVirtualenvOperator(
task_id="execute_python_in_venv",
python_callable=my_virtualenv_callable,
requirements=[
"pandas==2.0.3"
],
system_site_packages=False
)
```
Example Log:
```
created virtual environment CPython3.12.9.final.0-64 in 469ms
[2025-07-02, 14:48:54 CEST] {process_utils.py:194} INFO - creator CPython3Posix(dest=/tmp/venv2wzq4au8, clear=False, no_vcs_ignore=False, global=False)
[2025-07-02, 14:48:54 CEST] {process_utils.py:194} INFO - seeder FromAppData(download=False, pip=bundle, via=copy, app_data_dir=/home/airflow/.local/share/virtualenv)
[2025-07-02, 14:48:54 CEST] {process_utils.py:194} INFO - added seed packages: pip==24.3.1
[2025-07-02, 14:48:54 CEST] {process_utils.py:194} INFO - activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
[2025-07-02, 14:48:54 CEST] {process_utils.py:186} INFO - Executing cmd: /tmp/venv2wzq4au8/bin/pip install -r /tmp/venv2wzq4au8/requirements.txt
[2025-07-02, 14:48:54 CEST] {process_utils.py:190} INFO - Output:
[2025-07-02, 14:48:55 CEST] {process_utils.py:194} INFO - Collecting pandas==2.0.3 (from -r /tmp/venv2wzq4au8/requirements.txt (line 1))
[2025-07-02, 14:48:55 CEST] {process_utils.py:194} INFO - Downloading pandas-2.0.3.tar.gz (5.3 MB)
[2025-07-02, 14:48:55 CEST] {process_utils.py:194} INFO - ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.3/5.3 MB 146.7 MB/s eta 0:00:00
[2025-07-02, 14:48:56 CEST] {process_utils.py:194} INFO - Installing build dependencies: started
[2025-07-02, 14:49:01 CEST] {process_utils.py:194} INFO - Installing build dependencies: finished with status 'done'
[2025-07-02, 14:49:01 CEST] {process_utils.py:194} INFO - Getting requirements to build wheel: started
[2025-07-02, 14:50:01 CEST] {process_utils.py:194} INFO - Getting requirements to build wheel: finished with status 'done'
[2025-07-02, 14:50:01 CEST] {process_utils.py:194} INFO - Preparing metadata (pyproject.toml): started
[2025-07-02, 14:50:02 CEST] {process_utils.py:194} INFO - Preparing metadata (pyproject.toml): finished with status 'done'
[2025-07-02, 14:50:02 CEST] {process_utils.py:194} INFO - Collecting numpy>=1.21.0 (from pandas==2.0.3->-r /tmp/venv2wzq4au8/requirements.txt (line 1))
[2025-07-02, 14:50:02 CEST] {process_utils.py:194} INFO - Downloading numpy-2.3.1-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (62 kB)
[2025-07-02, 14:50:02 CEST] {process_utils.py:194} INFO - Collecting python-dateutil>=2.8.2 (from pandas==2.0.3->-r /tmp/venv2wzq4au8/requirements.txt (line 1))
[2025-07-02, 14:50:02 CEST] {process_utils.py:194} INFO - Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
[2025-07-02, 14:50:02 CEST] {process_utils.py:194} INFO - Collecting pytz>=2020.1 (from pandas==2.0.3->-r /tmp/venv2wzq4au8/requirements.txt (line 1))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add uv venv implementation [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #50999:
URL: https://github.com/apache/airflow/issues/50999#issuecomment-3024957249
Actually - itls already there https://airflow.apache.org/docs/apache-airflow-providers-standard/stable/configurations-ref.html#venv-install-method in the standard provider
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add uv venv implementation [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #50999:
URL: https://github.com/apache/airflow/issues/50999#issuecomment-2903992938
Sure - feel free to add it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add uv venv implementation [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #50999:
URL: https://github.com/apache/airflow/issues/50999#issuecomment-3028054118
Likely you should see somethig. You need to debug it - see if you have good standard provider installed (this is where Python Virtualenv has been moved to recently - look at standard provider when this config param has been added and see if your workers actually receive that configuration change you added. All that needs to be debugged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org