You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "romansivolobtsev (via GitHub)" <gi...@apache.org> on 2023/08/25 15:52:07 UTC

[GitHub] [airflow] romansivolobtsev opened a new issue, #33733: Template rendering doesn't work well with expand_kwargs

romansivolobtsev opened a new issue, #33733:
URL: https://github.com/apache/airflow/issues/33733

   ### Apache Airflow version
   
   2.7.0
   
   ### What happened
   
   I'm trying to make dynamic mapping for my operator which contains template variables. It works perfect with `.expand`, but there some strange issue with `.expand_kwargs`
   
   String variables passed from `expand` renders nice in all cases. But variables passed from `expand_kwargs` renders only if I change them somehow. Let's see the example.
   
   ```from datetime import datetime
   
   from airflow.decorators import dag
   from airflow.utils.task_group import TaskGroup
   from airflow.models import BaseOperator
   
   
   JINJA_VAR = '{{ data_interval_start.to_date_string() }}'
   
   
   class MyOperator(BaseOperator):
       template_fields = ('a', 'b')
   
       def __init__(self, x: str, **kwargs):
           super().__init__(**kwargs)
   
           self.a = x          # same objects doesn't render with expand_kwargs
           self.b = x + '!'    # new one always render
   
       def execute(self, context):
           self.log.info('$' * 300)
           self.log.info('a: ' + self.a)
           self.log.info('b: ' + self.b)
           self.log.info('%' * 300)
   
   
   def expand_operator(i=3):
       return (
           MyOperator
               .partial(task_id='expand')
               .expand(x=[JINJA_VAR for _ in range(i)])
       )
   
   
   def expand_kwargs_operator(i=3):
       return (
           MyOperator
               .partial(task_id='expand_kwargs')
               .expand_kwargs([{'x': JINJA_VAR} for _ in range(i)])
       )
   
   
   @dag(
       dag_id='expand_test',
       start_date=datetime.today(),
       schedule_interval='10 0 * * *'
   )
   def get_dag():
       with TaskGroup(group_id='task_group'):
           expand_operator(3), expand_kwargs_operator(4)
   
   dag = get_dag()
   ``` 
   
   ![Screenshot from 2023-08-25 18-29-25](https://github.com/apache/airflow/assets/35471502/b204fd4b-c8b0-413d-84cb-b89db5773071)
   ![Screenshot from 2023-08-25 18-29-39](https://github.com/apache/airflow/assets/35471502/d3b3e24f-6b8a-49f2-924f-f226ecc5447c)
   
   
   I tried airflow versions: 2.5.0, 2.5.3, 2.7.0
   
   ### What you think should happen instead
   
   `expand` and `expand_kwargs` logic should be the same for rendered templates
   
   ### How to reproduce
   
   Run my example dag
   
   ### Operating System
   
   ubuntu 20.04
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Docker-Compose
   
   ### Deployment details
   
   Tested docker images:
   
   - apache/airflow:2.5.0-python3.10
   - apache/airflow:2.5.3-python3.10
   - apache/airflow:2.7.0-python3.10
   
   CeleryExecutor with Redis and PostgreSQL
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Template rendering doesn't work well with expand_kwargs [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #33733: Template rendering doesn't work well with expand_kwargs
URL: https://github.com/apache/airflow/issues/33733


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Template rendering doesn't work well with expand_kwargs [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #33733:
URL: https://github.com/apache/airflow/issues/33733#issuecomment-2635849393

   This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] boring-cyborg[bot] commented on issue #33733: Template rendering doesn't work well with expand_kwargs

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #33733:
URL: https://github.com/apache/airflow/issues/33733#issuecomment-1693573441

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Template rendering doesn't work well with expand_kwargs [airflow]

Posted by "shahulp (via GitHub)" <gi...@apache.org>.
shahulp commented on issue #33733:
URL: https://github.com/apache/airflow/issues/33733#issuecomment-1906895041

   @romansivolobtsev I am curious if you had any work around for expand_kwargs or it doesn't work always. I assume you had workaround from this "renders only if I change them somehow" TIA


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Template rendering doesn't work well with expand_kwargs [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #33733:
URL: https://github.com/apache/airflow/issues/33733#issuecomment-2723816795

   This issue has been closed because it has not received response from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] peloyeje commented on issue #33733: Template rendering doesn't work well with expand_kwargs

Posted by "peloyeje (via GitHub)" <gi...@apache.org>.
peloyeje commented on issue #33733:
URL: https://github.com/apache/airflow/issues/33733#issuecomment-1732419699

   Hey, interesting, I can try to fix this one


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Template rendering doesn't work well with expand_kwargs [airflow]

Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #33733:
URL: https://github.com/apache/airflow/issues/33733#issuecomment-1746305972

   @peloyeje assigned you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org