You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jens-scheffler-bosch (via GitHub)" <gi...@apache.org> on 2023/09/30 17:04:20 UTC

[GitHub] [airflow] jens-scheffler-bosch commented on issue #34688: ExternalPythonOperator importing an Airflow library results in DAG terminating error

jens-scheffler-bosch commented on issue #34688:
URL: https://github.com/apache/airflow/issues/34688#issuecomment-1741814891

   To make a regression I tested on latest main and the version tag 2.7.1 with the following example DAG that installs the named venv inside a DAG preventing that I need to re-build exactly your Docker environment. With this setup I can not reproduce the error. Can you try on your side?
   
   ```
   from __future__ import annotations
   
   from datetime import datetime
   
   from airflow.decorators import dag, task
   from airflow.operators.bash import BashOperator
   
   
   @dag(
       schedule=None,
       start_date=datetime(2021, 1, 1),
   )
   def example_break():
       setup_external_python = BashOperator(
           task_id="setup_external_python",
           bash_command="python -m venv /opt/airflow/python_venvs/operators_venv"
       )
       install_packages = BashOperator(
           task_id="install_packages",
           bash_command="/opt/airflow/python_venvs/operators_venv/bin/pip install apache-airflow[postgres]==2.7.1"
       )
   
       @task.external_python(python="/opt/airflow/python_venvs/operators_venv/bin/python", expect_airflow=True)
       def this_should_break():
           from airflow.providers.postgres.hooks.postgres import PostgresHook
           print(f"I get some postgress hook object here: {PostgresHook()}")
           return "yes"
   
       cleanup_python = BashOperator(
           task_id="cleanup_python",
           bash_command="rm -Rvf /opt/airflow/python_venvs/operators_venv"
       )
   
       setup_external_python.as_setup() >> install_packages.as_setup() >> this_should_break() >> cleanup_python.as_teardown()
   
   example_break()
   ```
   
   I assume you setup difference is somewhat related to the creation of your venv. Can you test on your side and use the example to find a difference to your prepared Docker setup?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org