You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "benbole (via GitHub)" <gi...@apache.org> on 2023/09/29 13:57:38 UTC

[GitHub] [airflow] benbole opened a new issue, #34688: ExternalPythonOperator importing an Airflow library results in DAG terminating error

benbole opened a new issue, #34688:
URL: https://github.com/apache/airflow/issues/34688

   ### Apache Airflow version
   
   2.7.1
   
   ### What happened
   
   Some modules I have written have dependency conflicts with Airflow's required Python environment, so I decided to use `ExternalPythonOperator`s and venvs as prescribed in the Airflow documentation. I am able to build the venv, build the Airflow Docker containers, and import all DAGs without issue, but when the Airflow executes the `ExternalPythonOperator` which attempts to import an Airflow library (such as `PostgresHook`) it results in error. 
   
   Some additional information that may be helpful:
   - I am able to successfully import Airflow libraries within the venv's Python Interpreter when executing it via command line
   - The DAG is able to successfully run when replacing the `ExternalPythonOperator` with a typical `@task` `PythonOperator`
   
   ### What you think should happen instead
   
   _No response_
   
   ### How to reproduce
   
   Below is the simplest example DAG/ExternalPythonOperator I was able to produce which still results in error. It is scrapped together based on a simplified version of my DAG, so there may be some unnecessary imports.
   
   The DAG:
    ```
    from __future__ import annotations
   
   import os
   import yaml
   from datetime import datetime
   
   from airflow.models import DAG
   from airflow.decorators import dag, task
   
   DAG_ID = "example_break"
   
   @task.external_python(python='/opt/airflow/python_venvs/operators_venv/bin/python', expect_airflow=True)
   def this_should_break():
       from operators.break_example import example_break
       return example_break()
   
   @dag(dag_id=DAG_ID,
       schedule=None,
       start_date=datetime(2021, 1, 1),
       catchup=False,
       tags=["example"],
       )
   def ExampleBreak():
       this_should_break()
   
   ExampleBreak()
    ```
    
    The task:
    ```
    from airflow.providers.postgres.hooks.postgres import PostgresHook #You can substitute any Airflow library here 
   
   def example_break():
       print('This should fail')
    ```
    
   
   ### Operating System
   
   PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" NAME="Debian GNU/Linux" VERSION_ID="11" VERSION="11 (bullseye)" VERSION_CODENAME=bullseye ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"
   
   ### Versions of Apache Airflow Providers
   
   Attached are the pip list for both my Airflow environment and my venv.
   [venv_pip_list.txt](https://github.com/apache/airflow/files/12766222/venv_pip_list.txt)
   [airflow_docker_pip_list.txt](https://github.com/apache/airflow/files/12766223/airflow_docker_pip_list.txt)
   
   
   ### Deployment
   
   Docker-Compose
   
   ### Deployment details
   
   I have customized the default docker-compose in order to use a custom Dockerfile for importing needed libraries. You can find both the docker-compose and the Dockerfile attached here.
   [Dockerfile.txt](https://github.com/apache/airflow/files/12766240/Dockerfile.txt)
   [docker-compose.txt](https://github.com/apache/airflow/files/12766248/docker-compose.txt)
   
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] ExternalPythonOperator importing an Airflow library results in DAG terminating error [airflow]

Posted by "benbole (via GitHub)" <gi...@apache.org>.
benbole commented on issue #34688:
URL: https://github.com/apache/airflow/issues/34688#issuecomment-1743306845

   I was able to confirm the above import worked. However, I modified the `pip install` statement from `pip install apache-airflow[postgres]==2.7.1` to `pip install apache-airflow==2.7.1` it resulted in the same previously seen error:
   
   ```
   from __future__ import annotations
   import os
   import yaml
   from datetime import datetime
   from airflow.models import DAG
   from airflow.operators.bash import BashOperator
   from airflow.decorators import dag, task
   
   @dag(
       dag_id='new_example_break',
       schedule=None,
       start_date=datetime(2021, 1, 1),
   )
   def example_break():
       setup_external_python = BashOperator(
           task_id="setup_external_python",
           bash_command="python -m venv /opt/airflow/python_venvs/operators_venv"
       )
       install_packages = BashOperator(
           task_id="install_packages",
           bash_command="/opt/airflow/python_venvs/operators_venv/bin/pip install apache-airflow==2.7.1"
       )
   
       @task.external_python(python="/opt/airflow/python_venvs/operators_venv/bin/python", expect_airflow=True)
       def this_should_break():
           from airflow.providers.postgres.hooks.postgres import PostgresHook
           print(f"I get some postgress hook object here: {PostgresHook()}")
           return "yes"
   
       cleanup_python = BashOperator(
           task_id="cleanup_python",
           bash_command="rm -Rvf /opt/airflow/python_venvs/operators_venv"
       )
   
       setup_external_python.as_setup() >> install_packages.as_setup() >> this_should_break() >> cleanup_python.as_teardown()
   
   example_break()
   ```
   
   Resulted in the following exception:
   ```
   bab250711cb1
   *** Found local files:
   ***   * /opt/airflow/logs/dag_id=new_example_break/run_id=manual__2023-10-02T16:05:43.285961+00:00/task_id=this_should_break/attempt=1.log
   [2023-10-02T16:06:38.136+0000] {taskinstance.py:1157} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: new_example_break.this_should_break manual__2023-10-02T16:05:43.285961+00:00 [queued]>
   [2023-10-02T16:06:38.150+0000] {taskinstance.py:1157} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: new_example_break.this_should_break manual__2023-10-02T16:05:43.285961+00:00 [queued]>
   [2023-10-02T16:06:38.152+0000] {taskinstance.py:1359} INFO - Starting attempt 1 of 1
   [2023-10-02T16:06:38.167+0000] {taskinstance.py:1380} INFO - Executing <Task(_PythonExternalDecoratedOperator): this_should_break> on 2023-10-02 16:05:43.285961+00:00
   [2023-10-02T16:06:38.176+0000] {standard_task_runner.py:57} INFO - Started process 488 to run task
   [2023-10-02T16:06:38.181+0000] {standard_task_runner.py:84} INFO - Running: ['***', 'tasks', 'run', 'new_example_break', 'this_should_break', 'manual__2023-10-02T16:05:43.285961+00:00', '--job-id', '1472', '--raw', '--subdir', 'DAGS_FOLDER/extraction_pipeline.py', '--cfg-path', '/tmp/tmpmmcj3jul']
   [2023-10-02T16:06:38.186+0000] {standard_task_runner.py:85} INFO - Job 1472: Subtask this_should_break
   [2023-10-02T16:06:38.252+0000] {task_command.py:415} INFO - Running <TaskInstance: new_example_break.this_should_break manual__2023-10-02T16:05:43.285961+00:00 [running]> on host bab250711cb1
   [2023-10-02T16:06:38.348+0000] {taskinstance.py:1660} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='***' AIRFLOW_CTX_DAG_ID='new_example_break' AIRFLOW_CTX_TASK_ID='this_should_break' AIRFLOW_CTX_EXECUTION_DATE='2023-10-02T16:05:43.285961+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2023-10-02T16:05:43.285961+00:00'
   [2023-10-02T16:06:39.150+0000] {process_utils.py:182} INFO - Executing cmd: /opt/***/python_venvs/operators_venv/bin/python /tmp/tmdb2r5pv98/script.py /tmp/tmdb2r5pv98/script.in /tmp/tmdb2r5pv98/script.out /tmp/tmdb2r5pv98/string_args.txt /tmp/tmdb2r5pv98/termination.log
   [2023-10-02T16:06:39.169+0000] {process_utils.py:186} INFO - Output:
   [2023-10-02T16:06:39.943+0000] {process_utils.py:190} INFO - Traceback (most recent call last):
   [2023-10-02T16:06:39.944+0000] {process_utils.py:190} INFO -   File "/tmp/tmdb2r5pv98/script.py", line 31, in <module>
   [2023-10-02T16:06:39.944+0000] {process_utils.py:190} INFO -     res = this_should_break(*arg_dict["args"], **arg_dict["kwargs"])
   [2023-10-02T16:06:39.945+0000] {process_utils.py:190} INFO -   File "/tmp/tmdb2r5pv98/script.py", line 26, in this_should_break
   [2023-10-02T16:06:39.946+0000] {process_utils.py:190} INFO -     from ***.providers.postgres.hooks.postgres import PostgresHook
   [2023-10-02T16:06:39.948+0000] {process_utils.py:190} INFO -   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
   [2023-10-02T16:06:39.949+0000] {process_utils.py:190} INFO -   File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked
   [2023-10-02T16:06:39.950+0000] {process_utils.py:190} INFO -   File "<frozen importlib._bootstrap>", line 914, in _find_spec
   [2023-10-02T16:06:39.951+0000] {process_utils.py:190} INFO -   File "<frozen importlib._bootstrap_external>", line 1407, in find_spec
   [2023-10-02T16:06:39.952+0000] {process_utils.py:190} INFO -   File "<frozen importlib._bootstrap_external>", line 1373, in _get_spec
   [2023-10-02T16:06:39.953+0000] {process_utils.py:190} INFO -   File "<frozen importlib._bootstrap_external>", line 1239, in __iter__
   [2023-10-02T16:06:39.954+0000] {process_utils.py:190} INFO -   File "<frozen importlib._bootstrap_external>", line 1227, in _recalculate
   [2023-10-02T16:06:39.954+0000] {process_utils.py:190} INFO -   File "<frozen importlib._bootstrap_external>", line 1223, in _get_parent_path
   [2023-10-02T16:06:39.955+0000] {process_utils.py:190} INFO - KeyError: '***'
   [2023-10-02T16:06:40.022+0000] {taskinstance.py:1935} ERROR - Task failed with exception
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.8/site-packages/airflow/decorators/base.py", line 221, in execute
       return_value = super().execute(context)
     File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 395, in execute
       return super().execute(context=serializable_context)
     File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 192, in execute
       return_value = self.execute_callable()
     File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 720, in execute_callable
       return self._execute_python_callable_in_subprocess(python_path, tmp_path)
     File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 463, in _execute_python_callable_in_subprocess
       raise AirflowException(error_msg) from None
   airflow.exceptions.AirflowException: Process returned non-zero exit status 1.
   '***'
   [2023-10-02T16:06:40.029+0000] {taskinstance.py:1398} INFO - Marking task as FAILED. dag_id=new_example_break, task_id=this_should_break, execution_date=20231002T160543, start_date=20231002T160638, end_date=20231002T160640
   [2023-10-02T16:06:40.045+0000] {standard_task_runner.py:104} ERROR - Failed to execute job 1472 for task this_should_break (Process returned non-zero exit status 1.
   '***'; 488)
   [2023-10-02T16:06:40.087+0000] {local_task_job_runner.py:228} INFO - Task exited with return code 1
   [2023-10-02T16:06:40.111+0000] {taskinstance.py:2776} INFO - 1 downstream tasks scheduled from follow-on schedule check
   
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] boring-cyborg[bot] commented on issue #34688: ExternalPythonOperator importing an Airflow library results in DAG terminating error

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #34688:
URL: https://github.com/apache/airflow/issues/34688#issuecomment-1740939632

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] ExternalPythonOperator importing an Airflow library results in DAG terminating error [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #34688: ExternalPythonOperator importing an Airflow library results in DAG terminating error
URL: https://github.com/apache/airflow/issues/34688


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] ExternalPythonOperator importing an Airflow library results in DAG terminating error [airflow]

Posted by "jens-scheffler-bosch (via GitHub)" <gi...@apache.org>.
jens-scheffler-bosch commented on issue #34688:
URL: https://github.com/apache/airflow/issues/34688#issuecomment-1743787645

   Okay, I did another regression with the DAG you responded. In deed it seems in you run your external python env with only `apache-airflow==2.7.1` then importing fails in a circular dependency. Which not fails if you set up the python env with `apache-airflow[postgres]==2.7.1`.
   
   Is there a reason that you are not installing the postgres provider if you want to call the `PostgresHook`?
   
   Probably finding the root of the circular import is a bit complex to trace down but installing the required postgres dependency seems to be a good workaround.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] jens-scheffler-bosch commented on issue #34688: ExternalPythonOperator importing an Airflow library results in DAG terminating error

Posted by "jens-scheffler-bosch (via GitHub)" <gi...@apache.org>.
jens-scheffler-bosch commented on issue #34688:
URL: https://github.com/apache/airflow/issues/34688#issuecomment-1741814891

   To make a regression I tested on latest main and the version tag 2.7.1 with the following example DAG that installs the named venv inside a DAG preventing that I need to re-build exactly your Docker environment. With this setup I can not reproduce the error. Can you try on your side?
   
   ```
   from __future__ import annotations
   
   from datetime import datetime
   
   from airflow.decorators import dag, task
   from airflow.operators.bash import BashOperator
   
   
   @dag(
       schedule=None,
       start_date=datetime(2021, 1, 1),
   )
   def example_break():
       setup_external_python = BashOperator(
           task_id="setup_external_python",
           bash_command="python -m venv /opt/airflow/python_venvs/operators_venv"
       )
       install_packages = BashOperator(
           task_id="install_packages",
           bash_command="/opt/airflow/python_venvs/operators_venv/bin/pip install apache-airflow[postgres]==2.7.1"
       )
   
       @task.external_python(python="/opt/airflow/python_venvs/operators_venv/bin/python", expect_airflow=True)
       def this_should_break():
           from airflow.providers.postgres.hooks.postgres import PostgresHook
           print(f"I get some postgress hook object here: {PostgresHook()}")
           return "yes"
   
       cleanup_python = BashOperator(
           task_id="cleanup_python",
           bash_command="rm -Rvf /opt/airflow/python_venvs/operators_venv"
       )
   
       setup_external_python.as_setup() >> install_packages.as_setup() >> this_should_break() >> cleanup_python.as_teardown()
   
   example_break()
   ```
   
   I assume you setup difference is somewhat related to the creation of your venv. Can you test on your side and use the example to find a difference to your prepared Docker setup?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] ExternalPythonOperator importing an Airflow library results in DAG terminating error [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #34688:
URL: https://github.com/apache/airflow/issues/34688#issuecomment-2461466330

   This issue has been closed because it has not received response from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org