You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "benbole (via GitHub)" <gi...@apache.org> on 2023/09/29 13:57:38 UTC
[GitHub] [airflow] benbole opened a new issue, #34688: ExternalPythonOperator importing an Airflow library results in DAG terminating error
benbole opened a new issue, #34688:
URL: https://github.com/apache/airflow/issues/34688
### Apache Airflow version
2.7.1
### What happened
Some modules I have written have dependency conflicts with Airflow's required Python environment, so I decided to use `ExternalPythonOperator`s and venvs as prescribed in the Airflow documentation. I am able to build the venv, build the Airflow Docker containers, and import all DAGs without issue, but when the Airflow executes the `ExternalPythonOperator` which attempts to import an Airflow library (such as `PostgresHook`) it results in error.
Some additional information that may be helpful:
- I am able to successfully import Airflow libraries within the venv's Python Interpreter when executing it via command line
- The DAG is able to successfully run when replacing the `ExternalPythonOperator` with a typical `@task` `PythonOperator`
### What you think should happen instead
_No response_
### How to reproduce
Below is the simplest example DAG/ExternalPythonOperator I was able to produce which still results in error. It is scrapped together based on a simplified version of my DAG, so there may be some unnecessary imports.
The DAG:
```
from __future__ import annotations
import os
import yaml
from datetime import datetime
from airflow.models import DAG
from airflow.decorators import dag, task
DAG_ID = "example_break"
@task.external_python(python='/opt/airflow/python_venvs/operators_venv/bin/python', expect_airflow=True)
def this_should_break():
from operators.break_example import example_break
return example_break()
@dag(dag_id=DAG_ID,
schedule=None,
start_date=datetime(2021, 1, 1),
catchup=False,
tags=["example"],
)
def ExampleBreak():
this_should_break()
ExampleBreak()
```
The task:
```
from airflow.providers.postgres.hooks.postgres import PostgresHook #You can substitute any Airflow library here
def example_break():
print('This should fail')
```
### Operating System
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" NAME="Debian GNU/Linux" VERSION_ID="11" VERSION="11 (bullseye)" VERSION_CODENAME=bullseye ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"
### Versions of Apache Airflow Providers
Attached are the pip list for both my Airflow environment and my venv.
[venv_pip_list.txt](https://github.com/apache/airflow/files/12766222/venv_pip_list.txt)
[airflow_docker_pip_list.txt](https://github.com/apache/airflow/files/12766223/airflow_docker_pip_list.txt)
### Deployment
Docker-Compose
### Deployment details
I have customized the default docker-compose in order to use a custom Dockerfile for importing needed libraries. You can find both the docker-compose and the Dockerfile attached here.
[Dockerfile.txt](https://github.com/apache/airflow/files/12766240/Dockerfile.txt)
[docker-compose.txt](https://github.com/apache/airflow/files/12766248/docker-compose.txt)
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ExternalPythonOperator importing an Airflow library results in DAG terminating error [airflow]
Posted by "benbole (via GitHub)" <gi...@apache.org>.
benbole commented on issue #34688:
URL: https://github.com/apache/airflow/issues/34688#issuecomment-1743306845
I was able to confirm the above import worked. However, I modified the `pip install` statement from `pip install apache-airflow[postgres]==2.7.1` to `pip install apache-airflow==2.7.1` it resulted in the same previously seen error:
```
from __future__ import annotations
import os
import yaml
from datetime import datetime
from airflow.models import DAG
from airflow.operators.bash import BashOperator
from airflow.decorators import dag, task
@dag(
dag_id='new_example_break',
schedule=None,
start_date=datetime(2021, 1, 1),
)
def example_break():
setup_external_python = BashOperator(
task_id="setup_external_python",
bash_command="python -m venv /opt/airflow/python_venvs/operators_venv"
)
install_packages = BashOperator(
task_id="install_packages",
bash_command="/opt/airflow/python_venvs/operators_venv/bin/pip install apache-airflow==2.7.1"
)
@task.external_python(python="/opt/airflow/python_venvs/operators_venv/bin/python", expect_airflow=True)
def this_should_break():
from airflow.providers.postgres.hooks.postgres import PostgresHook
print(f"I get some postgress hook object here: {PostgresHook()}")
return "yes"
cleanup_python = BashOperator(
task_id="cleanup_python",
bash_command="rm -Rvf /opt/airflow/python_venvs/operators_venv"
)
setup_external_python.as_setup() >> install_packages.as_setup() >> this_should_break() >> cleanup_python.as_teardown()
example_break()
```
Resulted in the following exception:
```
bab250711cb1
*** Found local files:
*** * /opt/airflow/logs/dag_id=new_example_break/run_id=manual__2023-10-02T16:05:43.285961+00:00/task_id=this_should_break/attempt=1.log
[2023-10-02T16:06:38.136+0000] {taskinstance.py:1157} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: new_example_break.this_should_break manual__2023-10-02T16:05:43.285961+00:00 [queued]>
[2023-10-02T16:06:38.150+0000] {taskinstance.py:1157} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: new_example_break.this_should_break manual__2023-10-02T16:05:43.285961+00:00 [queued]>
[2023-10-02T16:06:38.152+0000] {taskinstance.py:1359} INFO - Starting attempt 1 of 1
[2023-10-02T16:06:38.167+0000] {taskinstance.py:1380} INFO - Executing <Task(_PythonExternalDecoratedOperator): this_should_break> on 2023-10-02 16:05:43.285961+00:00
[2023-10-02T16:06:38.176+0000] {standard_task_runner.py:57} INFO - Started process 488 to run task
[2023-10-02T16:06:38.181+0000] {standard_task_runner.py:84} INFO - Running: ['***', 'tasks', 'run', 'new_example_break', 'this_should_break', 'manual__2023-10-02T16:05:43.285961+00:00', '--job-id', '1472', '--raw', '--subdir', 'DAGS_FOLDER/extraction_pipeline.py', '--cfg-path', '/tmp/tmpmmcj3jul']
[2023-10-02T16:06:38.186+0000] {standard_task_runner.py:85} INFO - Job 1472: Subtask this_should_break
[2023-10-02T16:06:38.252+0000] {task_command.py:415} INFO - Running <TaskInstance: new_example_break.this_should_break manual__2023-10-02T16:05:43.285961+00:00 [running]> on host bab250711cb1
[2023-10-02T16:06:38.348+0000] {taskinstance.py:1660} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='***' AIRFLOW_CTX_DAG_ID='new_example_break' AIRFLOW_CTX_TASK_ID='this_should_break' AIRFLOW_CTX_EXECUTION_DATE='2023-10-02T16:05:43.285961+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2023-10-02T16:05:43.285961+00:00'
[2023-10-02T16:06:39.150+0000] {process_utils.py:182} INFO - Executing cmd: /opt/***/python_venvs/operators_venv/bin/python /tmp/tmdb2r5pv98/script.py /tmp/tmdb2r5pv98/script.in /tmp/tmdb2r5pv98/script.out /tmp/tmdb2r5pv98/string_args.txt /tmp/tmdb2r5pv98/termination.log
[2023-10-02T16:06:39.169+0000] {process_utils.py:186} INFO - Output:
[2023-10-02T16:06:39.943+0000] {process_utils.py:190} INFO - Traceback (most recent call last):
[2023-10-02T16:06:39.944+0000] {process_utils.py:190} INFO - File "/tmp/tmdb2r5pv98/script.py", line 31, in <module>
[2023-10-02T16:06:39.944+0000] {process_utils.py:190} INFO - res = this_should_break(*arg_dict["args"], **arg_dict["kwargs"])
[2023-10-02T16:06:39.945+0000] {process_utils.py:190} INFO - File "/tmp/tmdb2r5pv98/script.py", line 26, in this_should_break
[2023-10-02T16:06:39.946+0000] {process_utils.py:190} INFO - from ***.providers.postgres.hooks.postgres import PostgresHook
[2023-10-02T16:06:39.948+0000] {process_utils.py:190} INFO - File "<frozen importlib._bootstrap>", line 991, in _find_and_load
[2023-10-02T16:06:39.949+0000] {process_utils.py:190} INFO - File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked
[2023-10-02T16:06:39.950+0000] {process_utils.py:190} INFO - File "<frozen importlib._bootstrap>", line 914, in _find_spec
[2023-10-02T16:06:39.951+0000] {process_utils.py:190} INFO - File "<frozen importlib._bootstrap_external>", line 1407, in find_spec
[2023-10-02T16:06:39.952+0000] {process_utils.py:190} INFO - File "<frozen importlib._bootstrap_external>", line 1373, in _get_spec
[2023-10-02T16:06:39.953+0000] {process_utils.py:190} INFO - File "<frozen importlib._bootstrap_external>", line 1239, in __iter__
[2023-10-02T16:06:39.954+0000] {process_utils.py:190} INFO - File "<frozen importlib._bootstrap_external>", line 1227, in _recalculate
[2023-10-02T16:06:39.954+0000] {process_utils.py:190} INFO - File "<frozen importlib._bootstrap_external>", line 1223, in _get_parent_path
[2023-10-02T16:06:39.955+0000] {process_utils.py:190} INFO - KeyError: '***'
[2023-10-02T16:06:40.022+0000] {taskinstance.py:1935} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/decorators/base.py", line 221, in execute
return_value = super().execute(context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 395, in execute
return super().execute(context=serializable_context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 192, in execute
return_value = self.execute_callable()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 720, in execute_callable
return self._execute_python_callable_in_subprocess(python_path, tmp_path)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 463, in _execute_python_callable_in_subprocess
raise AirflowException(error_msg) from None
airflow.exceptions.AirflowException: Process returned non-zero exit status 1.
'***'
[2023-10-02T16:06:40.029+0000] {taskinstance.py:1398} INFO - Marking task as FAILED. dag_id=new_example_break, task_id=this_should_break, execution_date=20231002T160543, start_date=20231002T160638, end_date=20231002T160640
[2023-10-02T16:06:40.045+0000] {standard_task_runner.py:104} ERROR - Failed to execute job 1472 for task this_should_break (Process returned non-zero exit status 1.
'***'; 488)
[2023-10-02T16:06:40.087+0000] {local_task_job_runner.py:228} INFO - Task exited with return code 1
[2023-10-02T16:06:40.111+0000] {taskinstance.py:2776} INFO - 1 downstream tasks scheduled from follow-on schedule check
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #34688: ExternalPythonOperator importing an Airflow library results in DAG terminating error
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #34688:
URL: https://github.com/apache/airflow/issues/34688#issuecomment-1740939632
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ExternalPythonOperator importing an Airflow library results in DAG terminating error [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #34688: ExternalPythonOperator importing an Airflow library results in DAG terminating error
URL: https://github.com/apache/airflow/issues/34688
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ExternalPythonOperator importing an Airflow library results in DAG terminating error [airflow]
Posted by "jens-scheffler-bosch (via GitHub)" <gi...@apache.org>.
jens-scheffler-bosch commented on issue #34688:
URL: https://github.com/apache/airflow/issues/34688#issuecomment-1743787645
Okay, I did another regression with the DAG you responded. In deed it seems in you run your external python env with only `apache-airflow==2.7.1` then importing fails in a circular dependency. Which not fails if you set up the python env with `apache-airflow[postgres]==2.7.1`.
Is there a reason that you are not installing the postgres provider if you want to call the `PostgresHook`?
Probably finding the root of the circular import is a bit complex to trace down but installing the required postgres dependency seems to be a good workaround.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jens-scheffler-bosch commented on issue #34688: ExternalPythonOperator importing an Airflow library results in DAG terminating error
Posted by "jens-scheffler-bosch (via GitHub)" <gi...@apache.org>.
jens-scheffler-bosch commented on issue #34688:
URL: https://github.com/apache/airflow/issues/34688#issuecomment-1741814891
To make a regression I tested on latest main and the version tag 2.7.1 with the following example DAG that installs the named venv inside a DAG preventing that I need to re-build exactly your Docker environment. With this setup I can not reproduce the error. Can you try on your side?
```
from __future__ import annotations
from datetime import datetime
from airflow.decorators import dag, task
from airflow.operators.bash import BashOperator
@dag(
schedule=None,
start_date=datetime(2021, 1, 1),
)
def example_break():
setup_external_python = BashOperator(
task_id="setup_external_python",
bash_command="python -m venv /opt/airflow/python_venvs/operators_venv"
)
install_packages = BashOperator(
task_id="install_packages",
bash_command="/opt/airflow/python_venvs/operators_venv/bin/pip install apache-airflow[postgres]==2.7.1"
)
@task.external_python(python="/opt/airflow/python_venvs/operators_venv/bin/python", expect_airflow=True)
def this_should_break():
from airflow.providers.postgres.hooks.postgres import PostgresHook
print(f"I get some postgress hook object here: {PostgresHook()}")
return "yes"
cleanup_python = BashOperator(
task_id="cleanup_python",
bash_command="rm -Rvf /opt/airflow/python_venvs/operators_venv"
)
setup_external_python.as_setup() >> install_packages.as_setup() >> this_should_break() >> cleanup_python.as_teardown()
example_break()
```
I assume you setup difference is somewhat related to the creation of your venv. Can you test on your side and use the example to find a difference to your prepared Docker setup?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ExternalPythonOperator importing an Airflow library results in DAG terminating error [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #34688:
URL: https://github.com/apache/airflow/issues/34688#issuecomment-2461466330
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org