You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ScriptPup (via GitHub)" <gi...@apache.org> on 2023/02/02 14:17:06 UTC
[GitHub] [airflow] ScriptPup opened a new issue, #29318: Return error when requirements fail to install
ScriptPup opened a new issue, #29318:
URL: https://github.com/apache/airflow/issues/29318
### Description
When the PythonVirtualenvOperator fails to install requirements, it shows something like this `
```
subprocess.CalledProcessError: Command '['/tmp/venvre9qziim/bin/pip', 'install', '-r', '/tmp/venvre9qziim/requirements.txt']' returned non-zero exit status 1.
```
Add additional information about why the `pip install -r requirements.txt` failed. The problem could be a missing dependancy, a dependancy failing to build in the container, a network failure, etc.
### Use case/motivation
Currently, troubleshooting the PythonVirtualenvOperator requirement install failures is a horrible slog of trial and error to figure out what the problem is; especially in an environment where there's no access to the airflow server directly and we only have the ability to push DAGs to a repo and have it sync.
Adding additional information about the actual error returned by pip when it attempts an install would be extremely helpful.
### Related issues
Not that I could find.
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #29318: Return error when requirements fail to install
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #29318:
URL: https://github.com/apache/airflow/issues/29318#issuecomment-1433269799
@arjunanan6 assigned to you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #29318: Return error when requirements fail to install
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #29318:
URL: https://github.com/apache/airflow/issues/29318#issuecomment-1413815891
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] arjunanan6 commented on issue #29318: Return error when requirements fail to install
Posted by "arjunanan6 (via GitHub)" <gi...@apache.org>.
arjunanan6 commented on issue #29318:
URL: https://github.com/apache/airflow/issues/29318#issuecomment-1432557858
I would like to take this one, if no one else is working on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29318: Return error when requirements fail to install
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29318:
URL: https://github.com/apache/airflow/issues/29318#issuecomment-1477936418
Yep. Looks good. And we cannot improve this realy by knowing "all" things that failed - this is something that might get improved by `pip` at some point in time, we can't do much about it. Closing,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] arjunanan6 commented on issue #29318: Return error when requirements fail to install
Posted by "arjunanan6 (via GitHub)" <gi...@apache.org>.
arjunanan6 commented on issue #29318:
URL: https://github.com/apache/airflow/issues/29318#issuecomment-1475183918
@ScriptPup Sorry it took ages on this one, but life happened :)
I spent a little time looking at this today and I wanted to get a little more context on the problem, because to me, it appears that the information is already available in the error output.
This is from my attempt to install a non-existent requirement:
```console
[2023-03-19, 09:30:01 UTC] {task_command.py:406} INFO - Running <TaskInstance: VENVTASK.virtualenv_classic scheduled__2023-03-19T09:20:00+00:00 [running]> on host 7007f5fd4d69
[2023-03-19, 09:30:01 UTC] {taskinstance.py:1557} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='***' AIRFLOW_CTX_DAG_ID='VENVTASK' AIRFLOW_CTX_TASK_ID='virtualenv_classic' AIRFLOW_CTX_EXECUTION_DATE='2023-03-19T09:20:00+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='scheduled__2023-03-19T09:20:00+00:00'
[2023-03-19, 09:30:01 UTC] {process_utils.py:179} INFO - Executing cmd: /usr/local/bin/python -m virtualenv /tmp/venv89mk8x6s --system-site-packages
[2023-03-19, 09:30:01 UTC] {process_utils.py:183} INFO - Output:
[2023-03-19, 09:30:01 UTC] {process_utils.py:187} INFO - created virtual environment CPython3.8.16.final.0-64 in 345ms
[2023-03-19, 09:30:01 UTC] {process_utils.py:187} INFO - creator CPython3Posix(dest=/tmp/venv89mk8x6s, clear=False, no_vcs_ignore=False, global=True)
[2023-03-19, 09:30:01 UTC] {process_utils.py:187} INFO - seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
[2023-03-19, 09:30:01 UTC] {process_utils.py:187} INFO - added seed packages: pip==23.0, setuptools==67.1.0, wheel==0.38.4
[2023-03-19, 09:30:01 UTC] {process_utils.py:187} INFO - activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
[2023-03-19, 09:30:01 UTC] {process_utils.py:179} INFO - Executing cmd: /tmp/venv89mk8x6s/bin/pip install -r /tmp/venv89mk8x6s/requirements.txt
[2023-03-19, 09:30:01 UTC] {process_utils.py:183} INFO - Output:
[2023-03-19, 09:30:02 UTC] {process_utils.py:187} INFO - ERROR: Could not find a version that satisfies the requirement colowsssrama==0.4.0 (from versions: none)
[2023-03-19, 09:30:02 UTC] {process_utils.py:187} INFO - ERROR: No matching distribution found for colowsssrama==0.4.0
[2023-03-19, 09:30:03 UTC] {process_utils.py:187} INFO -
[2023-03-19, 09:30:03 UTC] {process_utils.py:187} INFO - [notice] A new release of pip is available: 23.0 -> 23.0.1
[2023-03-19, 09:30:03 UTC] {process_utils.py:187} INFO - [notice] To update, run: python -m pip install --upgrade pip
[2023-03-19, 09:30:03 UTC] {taskinstance.py:1837} ERROR - Task failed with exception
Traceback (most recent call last):
File "/opt/airflow/airflow/operators/python.py", line 356, in execute
return super().execute(context=serializable_context)
File "/opt/airflow/airflow/operators/python.py", line 176, in execute
return_value = self.execute_callable()
File "/opt/airflow/airflow/operators/python.py", line 540, in execute_callable
prepare_virtualenv(
File "/opt/airflow/airflow/utils/python_virtualenv.py", line 99, in prepare_virtualenv
execute_in_subprocess(pip_cmd)
File "/opt/airflow/airflow/utils/process_utils.py", line 168, in execute_in_subprocess
execute_in_subprocess_with_kwargs(cmd, cwd=cwd)
File "/opt/airflow/airflow/utils/process_utils.py", line 191, in execute_in_subprocess_with_kwargs
raise subprocess.CalledProcessError(exit_code, cmd)
subprocess.CalledProcessError: Command '['/tmp/venv89mk8x6s/bin/pip', 'install', '-r', '/tmp/venv89mk8x6s/requirements.txt']' returned non-zero exit status 1.
```
As you see here, before the output from CalledProcessError, there is sufficient INFO level logging available for the things that did go wrong:
```console
[2023-03-19, 09:30:02 UTC] {process_utils.py:187} INFO - ERROR: No matching distribution found for colowsssrama==0.4.0
```
From what I am looking at, your requirement is already implemented in process_utils. If you have multiple requirements, then it is a bit tricky because it fails at the very first one which could not be installed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #29318: Return error when requirements fail to install
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #29318: Return error when requirements fail to install
URL: https://github.com/apache/airflow/issues/29318
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] arjunanan6 commented on issue #29318: Return error when requirements fail to install
Posted by "arjunanan6 (via GitHub)" <gi...@apache.org>.
arjunanan6 commented on issue #29318:
URL: https://github.com/apache/airflow/issues/29318#issuecomment-1483295551
No worries, I took ages to get to it too. Happy to help :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ScriptPup commented on issue #29318: Return error when requirements fail to install
Posted by "ScriptPup (via GitHub)" <gi...@apache.org>.
ScriptPup commented on issue #29318:
URL: https://github.com/apache/airflow/issues/29318#issuecomment-1483245907
Very sorry, I've been away and while I saw the emails come in I wasn't in a position to check on what you were saying.
I checked our instance and found you're right... The error comes before the stacktrace with the failure. I must have missed that previously somehow. Sorry about that, thank you for checking on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org