You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "romansivolobtsev (via GitHub)" <gi...@apache.org> on 2023/08/25 15:52:07 UTC
[GitHub] [airflow] romansivolobtsev opened a new issue, #33733: Template rendering doesn't work well with expand_kwargs
romansivolobtsev opened a new issue, #33733:
URL: https://github.com/apache/airflow/issues/33733
### Apache Airflow version
2.7.0
### What happened
I'm trying to make dynamic mapping for my operator which contains template variables. It works perfect with `.expand`, but there some strange issue with `.expand_kwargs`
String variables passed from `expand` renders nice in all cases. But variables passed from `expand_kwargs` renders only if I change them somehow. Let's see the example.
```from datetime import datetime
from airflow.decorators import dag
from airflow.utils.task_group import TaskGroup
from airflow.models import BaseOperator
JINJA_VAR = '{{ data_interval_start.to_date_string() }}'
class MyOperator(BaseOperator):
template_fields = ('a', 'b')
def __init__(self, x: str, **kwargs):
super().__init__(**kwargs)
self.a = x # same objects doesn't render with expand_kwargs
self.b = x + '!' # new one always render
def execute(self, context):
self.log.info('$' * 300)
self.log.info('a: ' + self.a)
self.log.info('b: ' + self.b)
self.log.info('%' * 300)
def expand_operator(i=3):
return (
MyOperator
.partial(task_id='expand')
.expand(x=[JINJA_VAR for _ in range(i)])
)
def expand_kwargs_operator(i=3):
return (
MyOperator
.partial(task_id='expand_kwargs')
.expand_kwargs([{'x': JINJA_VAR} for _ in range(i)])
)
@dag(
dag_id='expand_test',
start_date=datetime.today(),
schedule_interval='10 0 * * *'
)
def get_dag():
with TaskGroup(group_id='task_group'):
expand_operator(3), expand_kwargs_operator(4)
dag = get_dag()
```


I tried airflow versions: 2.5.0, 2.5.3, 2.7.0
### What you think should happen instead
`expand` and `expand_kwargs` logic should be the same for rendered templates
### How to reproduce
Run my example dag
### Operating System
ubuntu 20.04
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
Tested docker images:
- apache/airflow:2.5.0-python3.10
- apache/airflow:2.5.3-python3.10
- apache/airflow:2.7.0-python3.10
CeleryExecutor with Redis and PostgreSQL
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Template rendering doesn't work well with expand_kwargs [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #33733: Template rendering doesn't work well with expand_kwargs
URL: https://github.com/apache/airflow/issues/33733
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Template rendering doesn't work well with expand_kwargs [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #33733:
URL: https://github.com/apache/airflow/issues/33733#issuecomment-2635849393
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #33733: Template rendering doesn't work well with expand_kwargs
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #33733:
URL: https://github.com/apache/airflow/issues/33733#issuecomment-1693573441
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Template rendering doesn't work well with expand_kwargs [airflow]
Posted by "shahulp (via GitHub)" <gi...@apache.org>.
shahulp commented on issue #33733:
URL: https://github.com/apache/airflow/issues/33733#issuecomment-1906895041
@romansivolobtsev I am curious if you had any work around for expand_kwargs or it doesn't work always. I assume you had workaround from this "renders only if I change them somehow" TIA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Template rendering doesn't work well with expand_kwargs [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #33733:
URL: https://github.com/apache/airflow/issues/33733#issuecomment-2723816795
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] peloyeje commented on issue #33733: Template rendering doesn't work well with expand_kwargs
Posted by "peloyeje (via GitHub)" <gi...@apache.org>.
peloyeje commented on issue #33733:
URL: https://github.com/apache/airflow/issues/33733#issuecomment-1732419699
Hey, interesting, I can try to fix this one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Template rendering doesn't work well with expand_kwargs [airflow]
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #33733:
URL: https://github.com/apache/airflow/issues/33733#issuecomment-1746305972
@peloyeje assigned you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org