You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/02/23 14:49:59 UTC
[GitHub] [airflow] ldealmei opened a new issue #14392: Refer Best Practices of using Variables in the Concepts/Variables section
ldealmei opened a new issue #14392:
URL: https://github.com/apache/airflow/issues/14392
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
**Description**
The `Variable` feature caused us some issue because of the performance impact it can have if badly used. See [here](https://issues.apache.org/jira/browse/AIRFLOW-29).
To a rather new user, it is not obvious that how the DAG is written can have a considerable impact in executing it.
In our case we were generating 100s of tasks each using `Variable.get` in its configuration, making loading the Dag extermely slow (and timeout).
**Use case / motivation**
Our suggestion would be that emphasis is put in the docs in the Concept/Variables section of this coupling between how Variable is used and performance/stability.
**Are you willing to submit a PR?**
I'll try to suggest something.
<!--- We accept contributions! -->
**Related Issues**
https://issues.apache.org/jira/browse/AIRFLOW-29
https://github.com/apache/airflow/issues/1380
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sharon2719 commented on issue #14392: Refer Best Practices of using Variables in the Concepts/Variables section
Posted by GitBox <gi...@apache.org>.
sharon2719 commented on issue #14392:
URL: https://github.com/apache/airflow/issues/14392#issuecomment-939270042
Can I take a shot at this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #14392: Refer Best Practices of using Variables in the Concepts/Variables section
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #14392:
URL: https://github.com/apache/airflow/issues/14392#issuecomment-939303453
Feel free !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #14392: Refer Best Practices of using Variables in the Concepts/Variables section
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #14392:
URL: https://github.com/apache/airflow/issues/14392#issuecomment-1027291977
> I can take a shot at improving this. I agree with @ldealmei that if you follow the example on the [Variables page](https://airflow.apache.org/docs/apache-airflow/stable/concepts/variables.html), it walks you straight into an antipattern that will slow the dag parsing down significantly.
The variable doc does mention:
"We also recommend that you try to keep most of your settings and configuration in your DAG files, so it can be versioned using source control; Variables are really only for values that are truly runtime-dependent."
Also we explain in:
https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#top-level-python-code
"Specifically you should not run any database access, heavy computations and networking operations."
I think the current doc is good enough. if someone think otherwise please explain (or better yet open a PR with your suggestion)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ldealmei commented on issue #14392: Refer Best Practices of using Variables in the Concepts/Variables section
Posted by GitBox <gi...@apache.org>.
ldealmei commented on issue #14392:
URL: https://github.com/apache/airflow/issues/14392#issuecomment-839557699
I am not sure what was unclear in my message. I believe I did suggest something and even articulated why it was relevant for me.
Would you like me to redact exactly what to change in the docs?
As I wrote, my suggestion is:
> Our suggestion would be that emphasis is put in the docs in the Concept/Variables section of this coupling between how Variable is used and performance/stability.
or as the title indicates, to simply:
> Refer Best Practices of using Variables in the Concepts/Variables section
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #14392: Refer Best Practices of using Variables in the Concepts/Variables section
Posted by GitBox <gi...@apache.org>.
eladkal closed issue #14392:
URL: https://github.com/apache/airflow/issues/14392
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #14392: Refer Best Practices of using Variables in the Concepts/Variables section
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #14392:
URL: https://github.com/apache/airflow/issues/14392#issuecomment-784255553
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] collinmcnulty commented on issue #14392: Refer Best Practices of using Variables in the Concepts/Variables section
Posted by GitBox <gi...@apache.org>.
collinmcnulty commented on issue #14392:
URL: https://github.com/apache/airflow/issues/14392#issuecomment-889881652
I can take a shot at improving this. I agree with @ldealmei that if you follow the example on the [Variables page](https://airflow.apache.org/docs/apache-airflow/stable/concepts/variables.html), it walks you straight into an antipattern that will slow the dag parsing down significantly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vikramkoka commented on issue #14392: Refer Best Practices of using Variables in the Concepts/Variables section
Posted by GitBox <gi...@apache.org>.
vikramkoka commented on issue #14392:
URL: https://github.com/apache/airflow/issues/14392#issuecomment-832120930
Please suggest something or even articulate the questions you would like answered in the documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org