You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/08 04:06:53 UTC
[GitHub] [airflow] smith-m opened a new issue #13559: Nested templated variables do not always render
smith-m opened a new issue #13559:
URL: https://github.com/apache/airflow/issues/13559
**Apache Airflow version**:
1.10.14 and 1.10.8.
**Environment**:
Python 3.6 and Airflow 1.10.14 on sqllite,
**What happened**:
Nested jinja templates do not consistently render when running tasks. TI run rendering behavior also differs from airflow UI and airflow render cli.
**What you expected to happen**:
Airflow should render nested jinja templates consistently and completely across each interface. Coming from airflow 1.8.2, this used to be the case.
<!-- What do you think went wrong? -->
This regression may have been introduced in 1.10.6 with a refactor of BaseOperator templating functionality.
https://github.com/apache/airflow/pull/5461
Whether or not a nested layer renders seems to differ based on which arg is being templated in an operator and perhaps order. Furthermore, it seems like the render cli and airflow ui each apply TI.render_templates() a second time, creating inconsistency in what nested templates get rendered.
There may be bug in the way BaseOperator.render_template() observes/caches templated fields
**How to reproduce it**:
From the most basic airflow setup
nested_template_bug.py
`
from datetime import datetime
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
with DAG('nested_template_bug', start_date=datetime(2021, 1, 1)) as dag:
arg0 = 'level_0_{{task.task_id}}_{{ds}}'
kwarg1 = 'level_1_{{task.op_args[0]}}'
def print_fields(arg0, kwarg1):
print(f'level 0 arg0: {arg0}')
print(f'level 1 kwarg1: {kwarg1}')
nested_render = PythonOperator(
task_id='nested_render',
python_callable=print_fields,
op_args=[arg0, ],
op_kwargs={
'kwarg1': kwarg1,
},
)
`
> airflow test nested_template_bug nested_render 2021-01-01
level 0 arg0: level_0_nested_render_2021-01-01
level 1 kwarg1: level_1_level_0_{{task.task_id}}_{{ds}}
> airflow render
# ----------------------------------------------------------
# property: op_args
# ----------------------------------------------------------
['level_0_nested_render_2020-12-01']
# ----------------------------------------------------------
# property: op_kwargs
# ----------------------------------------------------------
{'kwarg1': 'level_1_level_0_nested_render_2020-12-01'}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RosterIn commented on issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
RosterIn commented on issue #13559:
URL: https://github.com/apache/airflow/issues/13559#issuecomment-756697289
I don't know if this is a bug or by design but there are many votes for this in https://stackoverflow.com/questions/44855949/make-custom-airflow-macros-expand-other-macros
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #13559:
URL: https://github.com/apache/airflow/issues/13559#issuecomment-756661128
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] smith-m edited a comment on issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
smith-m edited a comment on issue #13559:
URL: https://github.com/apache/airflow/issues/13559#issuecomment-756909307
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] utkarshgupta137 commented on issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
utkarshgupta137 commented on issue #13559:
URL: https://github.com/apache/airflow/issues/13559#issuecomment-922331197
I was having a similar problem: I wanted to pass a templated argument to a function, but it kept getting passed as raw template instead of rendered one.
The solution is to call the macro inside a pythonoperator & pass the data using xcoms.
```
def get_date(**kwargs):
ds = kwargs["ds_no_dash"]
# {{ dag_run.conf.get("date", "{{ ds_no_dash }}") }} doesn't work
date = kwargs["dag_run"].conf.get("date", ds)
kwargs["ti"].xcom_push("date", base_data)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] XD-DENG commented on issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
XD-DENG commented on issue #13559:
URL: https://github.com/apache/airflow/issues/13559#issuecomment-762135039
I agree with @turbaszek on the potential complexity of "fixing" this.
What I'm thinking is: if we can have different method (other than templating) to achieve the same functionality/purpose, we don't have to bother with it. Correct me if I'm wrong please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #13559:
URL: https://github.com/apache/airflow/issues/13559#issuecomment-762092857
> One possibility here is using the previous state of each attribute and rendering each templated_field until all fields are stable.
I'm wondering how this may impact performance. Maybe it would be worth to have proof of concept?
WDYT @kaxil @XD-DENG ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] smith-m commented on issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
smith-m commented on issue #13559:
URL: https://github.com/apache/airflow/issues/13559#issuecomment-756909307
One possibility here is tracking the previous state of each attribute and rendering each templated_field until all fields are stable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #13559:
URL: https://github.com/apache/airflow/issues/13559#issuecomment-756661128
This seems to be also issue in 2.0:
```
[2021-01-08 09:46:58,236] {logging_mixin.py:103} INFO - level 0 arg0: level_0_nested_render_2021-01-08
[2021-01-08 09:46:58,239] {logging_mixin.py:103} INFO - level 1 kwarg1: level_1_level_0_{{task.task_id}}_{{ds}}
```
but it is consistent with rendered values in web ui:
<img width="636" alt="Screenshot 2021-01-08 at 10 51 02" src="https://user-images.githubusercontent.com/9528307/104001069-691b9f80-519f-11eb-9ab3-80c98677632e.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RosterIn commented on issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
RosterIn commented on issue #13559:
URL: https://github.com/apache/airflow/issues/13559#issuecomment-756697289
I don't know if this is a bug or by design but there are many votes for this in https://stackoverflow.com/questions/44855949/make-custom-airflow-macros-expand-other-macros
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #13559:
URL: https://github.com/apache/airflow/issues/13559#issuecomment-756701047
> https://stackoverflow.com/questions/44855949/make-custom-airflow-macros-expand-other-macros
I'm not sure if this is connected. From what I understand the problem is that `op_kwargs` are not rendered while they are templated field:
```
[2021-01-08 11:15:09,294] {python.py:112} INFO - op_kwargs: {'kwarg1': 'level_1_level_0_{{task.task_id}}_{{ds}}'}, op_args: ['level_0_nested_render_2021-01-08']
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] smith-m edited a comment on issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
smith-m edited a comment on issue #13559:
URL: https://github.com/apache/airflow/issues/13559#issuecomment-756909307
One possibility here is using the previous state of each attribute and rendering each templated_field until all fields are stable.
This will not address macro expansion in the stackoverflow example, but would support nested jinja
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] utkarshgupta137 edited a comment on issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
utkarshgupta137 edited a comment on issue #13559:
URL: https://github.com/apache/airflow/issues/13559#issuecomment-922331197
I was having a similar problem: I wanted to pass a templated argument to a function, but it kept getting passed as raw template instead of rendered one.
The solution is to call the macro inside a pythonoperator & pass the data using xcoms.
```
def get_date(**kwargs):
# {{ dag_run.conf.get("date", "{{ ds_no_dash }}") }} doesn't work
ds = kwargs["ds_no_dash"]
date = kwargs["dag_run"].conf.get("date", ds)
kwargs["ti"].xcom_push("date", base_data)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #13559:
URL: https://github.com/apache/airflow/issues/13559#issuecomment-926247149
If we support nested rendering we’d also need to worry about circular reference, which would require a significantly more elaborated algorithm to properly handle. Dependency resolution is hell, don’t go there if at all possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek removed a comment on issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
turbaszek removed a comment on issue #13559:
URL: https://github.com/apache/airflow/issues/13559#issuecomment-756701047
> https://stackoverflow.com/questions/44855949/make-custom-airflow-macros-expand-other-macros
I'm not sure if this is connected. From what I understand the problem is that `op_kwargs` are not rendered while they are templated field:
```
[2021-01-08 11:15:09,294] {python.py:112} INFO - op_kwargs: {'kwarg1': 'level_1_level_0_{{task.task_id}}_{{ds}}'}, op_args: ['level_0_nested_render_2021-01-08']
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13559:
URL: https://github.com/apache/airflow/issues/13559#issuecomment-927099212
I looked at this and the problem turned out to be a bit different.
True - we should not even attempt to solve the recursive rendering of fields cross-referencing each other. But this was not the case. This was a `kwargs` field which referred `opargs` field - and `kwargs` was AFTER `opargs` in the list of templated fields, so it **should** work in principle, without resolving the recursive problem.
I looked at it and it turned out, this was a problem introduced by #8805 - where instead of the original task we started to use a `copy` of the task locked for execution. Unfortunately - the task in `context` was still the original one, not the copy, which caused the rendering problem (as rendered fields in the 'context' task were not updated at all.
This means that if someone would use the ``context.task`` latet in Python Callable or custom operator, the fields were not rendered there either 😱 .
I fixed that in the #18516 by replacing the task in context with the same locked-down copy - not sure if this might have other side effects (but I think it could only be positive side effects :D ).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13559:
URL: https://github.com/apache/airflow/issues/13559#issuecomment-927099827
With the change from #18516, I got this:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] smith-m commented on issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
smith-m commented on issue #13559:
URL: https://github.com/apache/airflow/issues/13559#issuecomment-756909307
One possibility here is tracking the previous state of each attribute and rendering each templated_field until all fields are stable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek removed a comment on issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
turbaszek removed a comment on issue #13559:
URL: https://github.com/apache/airflow/issues/13559#issuecomment-756701047
> https://stackoverflow.com/questions/44855949/make-custom-airflow-macros-expand-other-macros
I'm not sure if this is connected. From what I understand the problem is that `op_kwargs` are not rendered while they are templated field:
```
[2021-01-08 11:15:09,294] {python.py:112} INFO - op_kwargs: {'kwarg1': 'level_1_level_0_{{task.task_id}}_{{ds}}'}, op_args: ['level_0_nested_render_2021-01-08']
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #13559:
URL: https://github.com/apache/airflow/issues/13559#issuecomment-927099212
I looked at this and the problem turned out to be a bit different.
True - we should not even attempt to solve the recursive rendering of fields cross-referencing each other. But this was not the case. This was a `kwargs` field which referred `opargs` field - and `kwargs` was AFTER `opargs` in the list of templated fields, so it **should** work in principle, without resolving the recursive problem.
I looked at it and it turned out, this was a problem introduced by #8805 - where instead of the original task we started to use a `copy` of the task locked for execution. Unfortunately - the task in `context` was still the original one, not the copy, which caused the rendering problem (as rendered fields in the 'context' task were not updated at all.
This means that if someone would use the ``context['task']`` later in Python Callable or custom operator, the fields were not rendered there either 😱 .
I fixed that in the #18516 by replacing the task in context with the same locked-down copy - not sure if this might have other side effects (but I think it could only be positive side effects :D ).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] smith-m edited a comment on issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
smith-m edited a comment on issue #13559:
URL: https://github.com/apache/airflow/issues/13559#issuecomment-756909307
One possibility here is tracking the previous state of each attribute and rendering each templated_field until all fields are stable.
This will not address macro expansion in the stackoverflow example, but would support nested jinja
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #13559:
URL: https://github.com/apache/airflow/issues/13559#issuecomment-756533449
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #13559:
URL: https://github.com/apache/airflow/issues/13559#issuecomment-756706981
I'm afraid that this is not simple to fix. The main problem I see is that template fields are rendered in the order they are defined in `op.template_fields`. This means that some values may reference values that were not yet rendered and this will cause a problem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #13559:
URL: https://github.com/apache/airflow/issues/13559
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #13559: Nested templated variables do not always render
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #13559:
URL: https://github.com/apache/airflow/issues/13559
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org