You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/11 08:16:07 UTC
[GitHub] [airflow] nshetty15 opened a new pull request #16926: modify best practices document to include the use of environment variables over airflow variables in top level DAG code
nshetty15 opened a new pull request #16926:
URL: https://github.com/apache/airflow/pull/16926
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #16926: Docs: Suggest use of Env vars instead of Airflow Vars in best practises doc
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #16926:
URL: https://github.com/apache/airflow/pull/16926
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #16926: modify best practices document to include the use of environment variables over airflow variables in top level DAG code
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #16926:
URL: https://github.com/apache/airflow/pull/16926#discussion_r667491445
##########
File path: docs/apache-airflow/best-practices.rst
##########
@@ -109,6 +109,9 @@ or if you need to deserialize a json object from the variable :
{{ var.json.<variable_name> }}
+Alternative option is to read from the environment variables in the top level python code instead of Airflow variables.
+This will avoid new connections to Airflow metadata DB everytime Airflow parses the python file.
Review comment:
From Airflow >= 1.10.10 -- you can use Environment Variables to set Airflow Variables, hence connections to DB are not made.
Docs: https://airflow.apache.org/docs/apache-airflow/stable/howto/variable.html#storing-variables-in-environment-variables
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #16926: modify best practices document to include the use of environment variables over airflow variables in top level DAG code
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #16926:
URL: https://github.com/apache/airflow/pull/16926#discussion_r667491445
##########
File path: docs/apache-airflow/best-practices.rst
##########
@@ -109,6 +109,9 @@ or if you need to deserialize a json object from the variable :
{{ var.json.<variable_name> }}
+Alternative option is to read from the environment variables in the top level python code instead of Airflow variables.
+This will avoid new connections to Airflow metadata DB everytime Airflow parses the python file.
Review comment:
From Airflow >= 1.10.10 -- you can use Environment Variables to set Airflow Variables, hence connections to DB are not made.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #16926: Docs: Suggest use of Env vars instead of Airflow Vars in best practises doc
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #16926:
URL: https://github.com/apache/airflow/pull/16926#issuecomment-877848396
Awesome work, congrats on your first merged pull request!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #16926: modify best practices document to include the use of environment variables over airflow variables in top level DAG code
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #16926:
URL: https://github.com/apache/airflow/pull/16926#issuecomment-877760310
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, itโs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better ๐.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #16926: modify best practices document to include the use of environment variables over airflow variables in top level DAG code
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #16926:
URL: https://github.com/apache/airflow/pull/16926#issuecomment-877765720
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nshetty15 commented on pull request #16926: modify best practices document to include the use of environment variables over airflow variables in top level DAG code
Posted by GitBox <gi...@apache.org>.
nshetty15 commented on pull request #16926:
URL: https://github.com/apache/airflow/pull/16926#issuecomment-877776893
> Unfortunately static check fail - I recommend installing pre-commit. You appear to have some extra whitespace in the doc.
Thanks! I have removed the trailing space and updated the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #16926: modify best practices document to include the use of environment variables over airflow variables in top level DAG code
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #16926:
URL: https://github.com/apache/airflow/pull/16926#discussion_r667491304
##########
File path: docs/apache-airflow/best-practices.rst
##########
@@ -109,6 +109,9 @@ or if you need to deserialize a json object from the variable :
{{ var.json.<variable_name> }}
+Alternative option is to read from the environment variables in the top level python code instead of Airflow variables.
+This will avoid new connections to Airflow metadata DB everytime Airflow parses the python file.
Review comment:
```suggestion
An alternative option is to use environment variables to manage Airflow Variables. This will avoid new connections to Airflow metadata DB every time Airflow parses the python file. For more information, see: :ref:`managing_variables`.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #16926: modify best practices document to include the use of environment variables over airflow variables in top level DAG code
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #16926:
URL: https://github.com/apache/airflow/pull/16926#issuecomment-877768159
Unfortunately static check fail - I recommend installing pre-commit. You appear to have some extra whitespace in the doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #16926: modify best practices document to include the use of environment variables over airflow variables in top level DAG code
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #16926:
URL: https://github.com/apache/airflow/pull/16926#discussion_r667464048
##########
File path: docs/apache-airflow/best-practices.rst
##########
@@ -109,6 +109,9 @@ or if you need to deserialize a json object from the variable :
{{ var.json.<variable_name> }}
+Alternative option is to read from the environment variables in the top level python code instead of airflow variables.
+This will avoid new connections to airflow metadata DB everytime Airflow parses the python file.
+
Review comment:
I think it's pretty understandable :D and it is right after, so I think I am happy with the way how it is now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a change in pull request #16926: modify best practices document to include the use of environment variables over airflow variables in top level DAG code
Posted by GitBox <gi...@apache.org>.
eladkal commented on a change in pull request #16926:
URL: https://github.com/apache/airflow/pull/16926#discussion_r667463452
##########
File path: docs/apache-airflow/best-practices.rst
##########
@@ -109,6 +109,9 @@ or if you need to deserialize a json object from the variable :
{{ var.json.<variable_name> }}
+Alternative option is to read from the environment variables in the top level python code instead of airflow variables.
+This will avoid new connections to airflow metadata DB everytime Airflow parses the python file.
+
Review comment:
I think it's better to place this text in the following paragraph "Top level Python Code" because you mention in the explanation "top level python code" but it's not defined till the user read the following paragraph. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a change in pull request #16926: modify best practices document to include the use of environment variables over airflow variables in top level DAG code
Posted by GitBox <gi...@apache.org>.
eladkal commented on a change in pull request #16926:
URL: https://github.com/apache/airflow/pull/16926#discussion_r667463452
##########
File path: docs/apache-airflow/best-practices.rst
##########
@@ -109,6 +109,9 @@ or if you need to deserialize a json object from the variable :
{{ var.json.<variable_name> }}
+Alternative option is to read from the environment variables in the top level python code instead of airflow variables.
+This will avoid new connections to airflow metadata DB everytime Airflow parses the python file.
+
Review comment:
I think it's better to place this text in the following paragraph "Top level Python Code" because you mention in the explanation "top level python code" but it's not really defined till the user read the following paragraph. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #16926: modify best practices document to include the use of environment variables over airflow variables in top level DAG code
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #16926:
URL: https://github.com/apache/airflow/pull/16926#discussion_r667491304
##########
File path: docs/apache-airflow/best-practices.rst
##########
@@ -109,6 +109,9 @@ or if you need to deserialize a json object from the variable :
{{ var.json.<variable_name> }}
+Alternative option is to read from the environment variables in the top level python code instead of Airflow variables.
+This will avoid new connections to Airflow metadata DB everytime Airflow parses the python file.
Review comment:
```suggestion
An alternative option is to use environment variables in the top-level python code or use Environment Variables to create and manage Airflow variables. to manage Airflow Variables. This will avoid new connections to Airflow metadata DB every time Airflow parses the python file. For more information, see: :ref:`managing_variables`.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org