You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "matthewblock (via GitHub)" <gi...@apache.org> on 2024/12/10 16:10:17 UTC
[I] Allow use of callable for `template_fields` for TaskFlow API tasks [airflow]
matthewblock opened a new issue, #44819:
URL: https://github.com/apache/airflow/issues/44819
### Description
PR #37028 in Airflow 2.10.0 introduced the ability to pass a Python callable to templateable fields, instead of a Jinja templated string:
https://www.astronomer.io/docs/learn/templating/#use-a-python-callable-for-template-fields
> In Airflow 2.10+ it is possible to pass a Python callable to templateable fields. This is especially useful when the parameter value is created using complex operations that might not be possible or are hard to read in Jinja.
However, this feature doesn't seem to work for `@task.*` decorated tasks.
IMO it doesn't make sense to introduce a new feature that only works with the old style of tasks.
This feature is to make the new Python callable templating work for `@task` decorated TaskFlow API tasks.
### Use case/motivation
I want to specify a Python function for a templated field, instead of a Jinja templated string, for the reasons mentioned in the linked PR. For complex logic, it's much easier to implement and read Jinja rendering functions than a Jinja templated string.
### Related issues
https://github.com/apache/airflow/pull/35844
https://github.com/apache/airflow/pull/37028
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow use of callable for `template_fields` for TaskFlow API tasks [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #44819: Allow use of callable for `template_fields` for TaskFlow API tasks
URL: https://github.com/apache/airflow/issues/44819
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow use of callable for `template_fields` for TaskFlow API tasks [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #44819:
URL: https://github.com/apache/airflow/issues/44819#issuecomment-2535512406
Task flow operators do not have templateable field at all - so it does not make sense to add it there. And in task flow task you can simply call a Python function from inside the decorated function - there is no need to pass callable to the taskflow obie t if you can call them directly. This request makes no sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow use of callable for `template_fields` for TaskFlow API tasks [airflow]
Posted by "matthewblock (via GitHub)" <gi...@apache.org>.
matthewblock commented on issue #44819:
URL: https://github.com/apache/airflow/issues/44819#issuecomment-2536407180
> Task flow operators do not have templateable field at all
Maybe my terminology is off, but if I decorate `PythonOperator`, the following are still templateable fields, because the user can specify a Jinja template string and it will get rendered by the time the task executes.
- `op_kwargs`
- `op_args`
- `templates_dict`
The first two technically can't be specified in a decorated task, instead Airflow is rolling up any "extra" args or kwargs the user specifies, but they still get rendered.
Here is a test DAG that demonstrates what I'm talking about - See the last task, which prints the un-rendered function.
```
from datetime import datetime, timedelta
from airflow.decorators import dag, task
default_args = {
"owner": "airflow",
"depends_on_past": False,
"retry_delay": timedelta(minutes=1),
"retries": 0,
}
@dag(
default_args=default_args,
start_date=datetime(2000, 1, 1),
catchup=False,
schedule=None,
)
def test_jinja_functions_dag():
def get_foo(context, jinja_env):
return context["ti"]
@task
def args_kwargs_task(*args, **kwargs):
print(args) # Prints tuple containing current date (ds)
print(kwargs["foo"]) # Prints Task Instance object (ti)
@task(templates_dict={"some_thing": "{{ ds }}"})
def templates_dict_class(**kwargs):
print(kwargs["templates_dict"]["some_thing"]) # Prints current date (ds)
@task
def direct_kwarg_task(foo):
print(foo) # Prints Task Instance object (ti)
@task
def direct_kwarg_no_rendering_task(foo):
print(foo) # Prints function object get_foo
(
args_kwargs_task("{{ ds }}", foo="{{ ti }}"),
templates_dict_class(),
direct_kwarg_task(foo="{{ ti }}"),
direct_kwarg_no_rendering_task(foo=get_foo),
)
test_jinja_functions_dag()
```
> And in task flow task you can simply call a Python function from inside the decorated function - there is no need to pass callable to the taskflow parameter if you can call them directly.
You can also do this in your `python_callable` in a non-decorated `PythonOperator` task 🤷♂️ but that's not the point. As a user, it doesn't make sense why I'm encouraged to use TaskFlow API going forward, but I can only use a new syntax introduced in 2.10.0 using the old style PythonOperator.
When I was developing my DAG, I was surprised that my template function didn't actually get called; my task didn't fail because my function outputted a string and instead the name of the function ```<function test_jinja_functions_dag.<***s>.get_foo at 0xffffacac0a40>``` was getting passed everywhere.
At the least I think the documentation should be updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow use of callable for `template_fields` for TaskFlow API tasks [airflow]
Posted by "topherinternational (via GitHub)" <gi...@apache.org>.
topherinternational commented on issue #44819:
URL: https://github.com/apache/airflow/issues/44819#issuecomment-2535223998
Could you provide a code snippet of a @task-decorated function where the behavior isn't working as you expect?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow use of callable for `template_fields` for TaskFlow API tasks [airflow]
Posted by "matthewblock (via GitHub)" <gi...@apache.org>.
matthewblock commented on issue #44819:
URL: https://github.com/apache/airflow/issues/44819#issuecomment-2539580141
@potiuk can I open this as a discussion? Maybe this doesn't warrant a feature request but IMO this behavior is unexpected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org