You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jens-scheffler-bosch (via GitHub)" <gi...@apache.org> on 2023/09/30 17:04:20 UTC
[GitHub] [airflow] jens-scheffler-bosch commented on issue #34688: ExternalPythonOperator importing an Airflow library results in DAG terminating error
jens-scheffler-bosch commented on issue #34688:
URL: https://github.com/apache/airflow/issues/34688#issuecomment-1741814891
To make a regression I tested on latest main and the version tag 2.7.1 with the following example DAG that installs the named venv inside a DAG preventing that I need to re-build exactly your Docker environment. With this setup I can not reproduce the error. Can you try on your side?
```
from __future__ import annotations
from datetime import datetime
from airflow.decorators import dag, task
from airflow.operators.bash import BashOperator
@dag(
schedule=None,
start_date=datetime(2021, 1, 1),
)
def example_break():
setup_external_python = BashOperator(
task_id="setup_external_python",
bash_command="python -m venv /opt/airflow/python_venvs/operators_venv"
)
install_packages = BashOperator(
task_id="install_packages",
bash_command="/opt/airflow/python_venvs/operators_venv/bin/pip install apache-airflow[postgres]==2.7.1"
)
@task.external_python(python="/opt/airflow/python_venvs/operators_venv/bin/python", expect_airflow=True)
def this_should_break():
from airflow.providers.postgres.hooks.postgres import PostgresHook
print(f"I get some postgress hook object here: {PostgresHook()}")
return "yes"
cleanup_python = BashOperator(
task_id="cleanup_python",
bash_command="rm -Rvf /opt/airflow/python_venvs/operators_venv"
)
setup_external_python.as_setup() >> install_packages.as_setup() >> this_should_break() >> cleanup_python.as_teardown()
example_break()
```
I assume you setup difference is somewhat related to the creation of your venv. Can you test on your side and use the example to find a difference to your prepared Docker setup?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org