You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/11 18:09:34 UTC
[GitHub] [airflow] potiuk opened a new issue #22188: Add `{{ kwargs }}` as template parameter
potiuk opened a new issue #22188:
URL: https://github.com/apache/airflow/issues/22188
### Description
I've seen recently at least two cases that the users wanted to use constructor kwargs values in jinja templates as part of the Jinja `Context`.
### Use case/motivation
While this is not a trivial thing (because of potentially recursive references - even going through a chain of dependencies), this seems as an appealing use case.
A good example (slightly modified: https://apache-airflow.slack.com/archives/CCR6P6JRL/p1647017726404589?thread_ts=1646967780.298339&cid=CCR6P6JRL) :
```
copy_into_table = SnowflakeOperator(
task_id='copy_into_table',
snowflake_conn_id='snflk_conn_id',
sql='load_abc.sql',
warehouse='main_xs_wh',
schema='DBT_BASE',
sql_params={
'snflk_schema':'DBT_BASE',
's3_path': '@public.sf_stage/airflow/wsq-extract/{{var.value.snflk_env_var}}/',
'file_format': 'public.skip_header',
'batch_id':batch_id
},
)
```
In the sql file
```
FROM {{kwargs.sql_params.s3_path}}{{ds.replace('-', '/')}}/{{execution_date.int_timestamp}}.csv.gz
(file_format=> {{kwargs.sql_params.file_format}})
```
This allows to easily customize the content of rendered JINJA parameters not only based on state of the dag or ti, but also on
the args of the Operator, which allows to reuse the operator more easily with different parameters relying on Airflow's Jinja engine and no-need to modify the code of existing operators, rather than writing extra execute() code to parse and process the parameters.
There are a few things that might be discussed here:
1) The {{ kwargs }} should only keep the original "non-templated" copy of the values of the kwargs. I.e. if there are some templated_fields there, the original `{{ jinja }}` value should be returned when kwarg is accessed (otherwise we are vulnerable to recursion or we start to depend on sequence of processing of the kwargs by the templating engine)
2) We should consider to have some "generic" kwarg name for BaseOperator ("extra" ?). Currently when we pass an unknown kwarg, to Base Operator vhen `operators/allow_illegal_arguments` config is set to True (which is default and disabling it wil be dropped in the future). Otherwise it will be difficult to "add" new parameters dynamically - for example above "sql_params" is an illegal parameter in `SnowflakeOperator` - and it would be nice to have a way to pass such "jinj" context values to all operators. We could define such kwarg name and always allow it to be set on any operator.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #22188: Add `{{ kwargs }}` as template parameter
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #22188:
URL: https://github.com/apache/airflow/issues/22188#issuecomment-1065363364
Silly me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #22188: Add `{{ kwargs }}` as template parameter
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #22188:
URL: https://github.com/apache/airflow/issues/22188#issuecomment-1065362352
Hmmm. Is this not possible already with `params` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #22188: Add `{{ kwargs }}` as template parameter
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #22188:
URL: https://github.com/apache/airflow/issues/22188#issuecomment-1065362352
Hmmm. Is this not possible already with `params` ? (I could have missed that entirely).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #22188: Add `{{ kwargs }}` as template parameter
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #22188:
URL: https://github.com/apache/airflow/issues/22188
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org