You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "erdos2n (via GitHub)" <gi...@apache.org> on 2023/03/10 14:13:34 UTC
[GitHub] [airflow] erdos2n opened a new issue, #30023: Variable with template is ambiguous, especially for new users
erdos2n opened a new issue, #30023:
URL: https://github.com/apache/airflow/issues/30023
### What do you see as an issue?
In the doc below, it states `Make sure to use variable with template in operator, not in the top level code.`
https://github.com/apache/airflow/blob/main/docs/apache-airflow/best-practices.rst
It then gives this example as a Good Example.
**Good Example**
```
bash_use_variable_good = BashOperator(
task_id="bash_use_variable_good",
bash_command="echo variable foo=${foo_env}",
env={"foo_env": "{{ var.value.get('foo') }}"},
)
```
example below, since `{{ var.value.get('foo') }}` is in the top level code (since the `__init__` method is run every time the dag file is parsed.
This can be ambiguous for users, especially new users, to understand the true difference between templated and non-templated variables.
The difference between the two examples below isn't that one of them is using top-level code and the other isn't, it's that one is jinja templated and the other isn't. There is a great opportunity here to showcase the utility of jinja templating.
```
bash_use_variable_bad_3 = BashOperator(
task_id="bash_use_variable_bad_3",
bash_command="echo variable foo=${foo_env}",
env={"foo_env": Variable.get("foo")}, # DON'T DO THAT
)
```
and
```
bash_use_variable_good = BashOperator(
task_id="bash_use_variable_good",
bash_command="echo variable foo=${foo_env}",
env={"foo_env": "{{ var.value.get('foo') }}"},
)
```
### Solving the problem
Replacing `Make sure to use variable with template in operator, not in the top level code.` with a sentence that is more in line with the examples following it will not only show alignment but also highlight the benefits of jinja templating in top level code.
Perhaps:
```
In top-level code, variables using jinja templates do not produce a request until runtime, whereas, `Variable.get()` produces a request every time the dag file is parsed by the scheduler. This will lead to suboptimal performance for the scheduler and can cause the dag file to timeout before it is fully parsed.
```
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] erdos2n commented on issue #30023: Variable with template is ambiguous, especially for new users
Posted by "erdos2n (via GitHub)" <gi...@apache.org>.
erdos2n commented on issue #30023:
URL: https://github.com/apache/airflow/issues/30023#issuecomment-1464024231
Thanks! Is it possible to define `top level code` somewhere in the docs? Is there a page of key terms?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on issue #30023: Variable with template is ambiguous, especially for new users
Posted by "josh-fell (via GitHub)" <gi...@apache.org>.
josh-fell commented on issue #30023:
URL: https://github.com/apache/airflow/issues/30023#issuecomment-1464015111
What is and what isn't "top-level code" is a perplexing concept for new users I've run into as well. All yours @erdos2n!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #30023: Variable with template is ambiguous, especially for new users
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #30023: Variable with template is ambiguous, especially for new users
URL: https://github.com/apache/airflow/issues/30023
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] erdos2n commented on issue #30023: Variable with template is ambiguous, especially for new users
Posted by "erdos2n (via GitHub)" <gi...@apache.org>.
erdos2n commented on issue #30023:
URL: https://github.com/apache/airflow/issues/30023#issuecomment-1464027338
There's top level code in Python and top level code in Airflow and I don't know if they're exactly equivalent. I think `top level code` in Airflow should have a definition we all agree on, something like
## Top Level Code
Any code that is executed during the dag file process is considered top level code.
Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] erdos2n commented on issue #30023: Variable with template is ambiguous, especially for new users
Posted by "erdos2n (via GitHub)" <gi...@apache.org>.
erdos2n commented on issue #30023:
URL: https://github.com/apache/airflow/issues/30023#issuecomment-1464915691
https://github.com/apache/airflow/pull/30040
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org