[D] Using a partial to save boilerplate when using @task.external_python generates a NameError [airflow]

["dpopowich-aca (via GitHub)" <gi...@apache.org>]

GitHub user dpopowich-aca created a discussion: Using a partial to save boilerplate when using @task.external_python generates a NameError

I'm aware that you cannot reference top-level module code when using `@task.external_python`, but I'm surprised by the following...

To avoid boilerplate and ensure consistency across our codebase, I've placed the following in `mylib`, our custom python library installed into the airflow venv:

```python
# mylib
import functools

from airflow.sdk import task

PATH_TO_PYTHON = '/path/to/our/python/on/worker'

mytask = functools.partial(task.external_python, python=PATH_TO_PYTHON)
```

Then in a dag source file:

```python
# a test dag
from airflow.sdk import dag
from mylib import mytask

@dag(...)
def mydag():
    @mytask
    def some_task():
        ...
    mytask()
mydag()
```

When I trigger this dag I get a `NameError: name 'mytask' is not defined`. 

Unfortunately, I can't see the temporary file on the worker because it gets cleaned up after executing (anyway of saving that for debugging?), but anecdotal evidence suggests that airflow includes the decorator in the generated source and bombs because `mytask` is unknown.

Is my only option explicitly using `@task.external_python(...)`. 

GitHub link: https://github.com/apache/airflow/discussions/61409

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Using a partial to save boilerplate when using @task.external_python generates a NameError [airflow]

["potiuk (via GitHub)" <gi...@apache.org>]

GitHub user potiuk added a comment to the discussion: Using a partial to save boilerplate when using @task.external_python generates a NameError

I think it would be helpful if you provide a stack trace , rather than just the error. But generally speaking you can explore provider's code and modify things in installed package to debug things. 


The virtualenv/external_python code and scripts generated can be seen here https://github.com/apache/airflow/tree/main/providers/standard/src/airflow/providers/standard/utils

GitHub link: https://github.com/apache/airflow/discussions/61409#discussioncomment-15687115

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Using a partial to save boilerplate when using @task.external_python generates a NameError [airflow]

["dpopowich-aca (via GitHub)" <gi...@apache.org>]

GitHub user dpopowich-aca edited a discussion: Using a partial to save boilerplate when using @task.external_python generates a NameError

I'm aware that you cannot reference top-level module code when using `@task.external_python`, but I'm surprised by the following...

To avoid boilerplate and ensure consistency across our codebase, I've placed the following in `mylib`, our custom python library installed into the airflow venv:

```python
# mylib
import functools

from airflow.sdk import task

PATH_TO_PYTHON = '/path/to/our/python/on/worker'

mytask = functools.partial(task.external_python, python=PATH_TO_PYTHON)
```

Then in a dag source file:

```python
# a test dag
from airflow.sdk import dag
from mylib import mytask

@dag(...)
def mydag():
    @mytask()
    def some_task():
        ...
    some_task()
mydag()
```

When I trigger this dag I get a `NameError: name 'mytask' is not defined`. 

Unfortunately, I can't see the temporary file on the worker because it gets cleaned up after executing (anyway of saving that for debugging?), but anecdotal evidence suggests that airflow includes the decorator in the generated source and bombs because `mytask` is unknown.

Is my only option explicitly using `@task.external_python(...)`. 

GitHub link: https://github.com/apache/airflow/discussions/61409

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org