You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "JCoder01 (via GitHub)" <gi...@apache.org> on 2024/02/01 18:31:04 UTC

[I] external_python_task decorator doesn't work if airflow isn't in external python env [airflow]

JCoder01 opened a new issue, #37136:
URL: https://github.com/apache/airflow/issues/37136

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### If "Other Airflow 2 version" selected, which one?
   
   2.7.3
   
   ### What happened?
   
   ```
   2024-02-01T15:01:41.731+0000] {process_utils.py:186} INFO - Output:
   [2024-02-01T15:01:41.912+0000] {process_utils.py:190} INFO - Traceback (most recent call last):
   [2024-02-01T15:01:41.912+0000] {process_utils.py:190} INFO -   File "/tmp/tmdfow99m6c/script.py", line 17, in <module>
   [2024-02-01T15:01:41.913+0000] {process_utils.py:190} INFO -     @external_python_task(task_id = 'task_runner', python=PYENV, expect_airflow=False)
   [2024-02-01T15:01:41.913+0000] {process_utils.py:190} INFO -      ^^^^^^^^^^^^^^^^^^^^
   [2024-02-01T15:01:41.913+0000] {process_utils.py:190} INFO - NameError: name 'external_python_task' is not defined
   ```
   
   When generating `python_callable_source` the airflow decorator is included in the source
   
   ### What you think should happen instead?
   
   Airflow should strip the decorator from the function definition
   
   ### How to reproduce
   
   Create python env that doesn't install airflow (PYENV)
   
   run something to effect of:
   
   ```python
   
   from airflow.decorators import external_python_task
   from airflow.models.taskinstance import TaskInstance
   
   @external_python_task(python=PYENV, expect_airflow=False)
   def task_runner():
       pass
   
   
   with DAG(dag_id="test", start_date=pendulum.now()):
       task_runner()
   
   task = dag.get_task('task_runner')
   t1 = TaskInstance(task=task, execution_date=dag.get_latest_execution_date())
   t1.render_templates()
   task.execute(t1.get_template_context())
   ```
   
   
   
   ### Operating System
   
   linux
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] external_python_task decorator doesn't work if airflow isn't in external python env [airflow]

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #37136:
URL: https://github.com/apache/airflow/issues/37136#issuecomment-1923567769

   You should use `@task.external_python` instead, the function which you use is a private
   
   https://github.com/apache/airflow/blob/3904206b69428525db31ff7813daa0322f7b83e8/airflow/decorators/external_python.py#L43-L49
   
   I can't reproduce it locally on Airflow 2.8 it works as expected
   
   ```python
   from datetime import datetime, timezone
   from pathlib import Path
   
   from airflow import DAG
   from airflow.decorators import task, external_python_task
   
   
   ANOTHER_VENV = Path("/Users/taragolis/.pyenv/versions/llama-runtime-py311/bin/python")
   assert ANOTHER_VENV.exists()
   assert ANOTHER_VENV.is_file()
   
   
   @task.external_python(python=str(ANOTHER_VENV), expect_airflow=False)
   def task_runner():
       import sys
   
       if sys.version_info < (3, 10):
           from importlib_metadata import version as metadata_version
       else:
           from importlib.metadata import version as metadata_version
   
       try:
           ver = metadata_version("apache-airflow")
       except ModuleNotFoundError:
           print("`apache-airflow` package not found")
       else:
           print(f"found `apache-airflow` package version {ver}")
   
       return sys.executable
   
   
   with DAG(
       dag_id="issue_37136",
       start_date=datetime(2024, 1, 1, tzinfo=timezone.utc),
       schedule=None,
       tags=["venv", "37136"],
   ):
       task_runner()
   
   ```
   
   ```console
   [2024-02-02, 10:57:31 UTC] {taskinstance.py:1957} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: issue_37136.task_runner manual__2024-02-02T10:57:28.849895+00:00 [queued]>
   [2024-02-02, 10:57:31 UTC] {taskinstance.py:1957} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: issue_37136.task_runner manual__2024-02-02T10:57:28.849895+00:00 [queued]>
   [2024-02-02, 10:57:31 UTC] {taskinstance.py:2171} INFO - Starting attempt 1 of 1
   [2024-02-02, 10:57:31 UTC] {taskinstance.py:2192} INFO - Executing <Task(_PythonExternalDecoratedOperator): task_runner> on 2024-02-02 10:57:28.849895+00:00
   [2024-02-02, 10:57:31 UTC] {standard_task_runner.py:60} INFO - Started process 35155 to run task
   [2024-02-02, 10:57:31 UTC] {standard_task_runner.py:87} INFO - Running: ['airflow', 'tasks', 'run', 'issue_37136', 'task_runner', 'manual__2024-02-02T10:57:28.849895+00:00', '--job-id', '17', '--raw', '--subdir', 'DAGS_FOLDER/foo.py', '--cfg-path', '/var/folders/g7/vy97v5m51_v9hdpf_qxqzqvr0000gn/T/tmp8tbefl9b']
   [2024-02-02, 10:57:31 UTC] {standard_task_runner.py:88} INFO - Job 17: Subtask task_runner
   [2024-02-02, 10:57:31 UTC] {task_command.py:423} INFO - Running <TaskInstance: issue_37136.task_runner manual__2024-02-02T10:57:28.849895+00:00 [running]> on host andreys-mbp.lan
   [2024-02-02, 10:57:31 UTC] {taskinstance.py:2481} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='issue_37136' AIRFLOW_CTX_TASK_ID='task_runner' AIRFLOW_CTX_EXECUTION_DATE='2024-02-02T10:57:28.849895+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2024-02-02T10:57:28.849895+00:00'
   [2024-02-02, 10:57:31 UTC] {process_utils.py:182} INFO - Executing cmd: /Users/taragolis/.pyenv/versions/llama-runtime-py311/bin/python /var/folders/g7/vy97v5m51_v9hdpf_qxqzqvr0000gn/T/venv-calld6yc0zua/script.py /var/folders/g7/vy97v5m51_v9hdpf_qxqzqvr0000gn/T/venv-calld6yc0zua/script.in /var/folders/g7/vy97v5m51_v9hdpf_qxqzqvr0000gn/T/venv-calld6yc0zua/script.out /var/folders/g7/vy97v5m51_v9hdpf_qxqzqvr0000gn/T/venv-calld6yc0zua/string_args.txt /var/folders/g7/vy97v5m51_v9hdpf_qxqzqvr0000gn/T/venv-calld6yc0zua/termination.log
   [2024-02-02, 10:57:31 UTC] {process_utils.py:186} INFO - Output:
   [2024-02-02, 10:57:31 UTC] {process_utils.py:190} INFO - `apache-airflow` package not found
   [2024-02-02, 10:57:31 UTC] {python.py:201} INFO - Done. Returned value was: /Users/taragolis/.pyenv/versions/llama-runtime-py311/bin/python
   [2024-02-02, 10:57:31 UTC] {taskinstance.py:1138} INFO - Marking task as SUCCESS. dag_id=issue_37136, task_id=task_runner, execution_date=20240202T105728, start_date=20240202T105731, end_date=20240202T105731
   [2024-02-02, 10:57:31 UTC] {local_task_job_runner.py:234} INFO - Task exited with return code 0
   [2024-02-02, 10:57:31 UTC] {taskinstance.py:3281} INFO - 0 downstream tasks scheduled from follow-on schedule check
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] external_python_task decorator doesn't work if airflow isn't in external python env [airflow]

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis closed issue #37136: external_python_task decorator doesn't work if airflow isn't in external python env
URL: https://github.com/apache/airflow/issues/37136


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org