You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/06 02:48:30 UTC
[GitHub] [airflow] mik-laj opened a new pull request #13502: Add docs about mocking variables and connections
mik-laj opened a new pull request #13502:
URL: https://github.com/apache/airflow/pull/13502
Some users are asking how to configure variables in tests without using a database, so I do add some documentation.
Slalck discussion: https://apache-airflow.slack.com/archives/CSS36QQS1/p1609874991010700
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #13502: Add docs about mocking variables and connections
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #13502:
URL: https://github.com/apache/airflow/pull/13502#discussion_r552487016
##########
File path: docs/apache-airflow/best-practices.rst
##########
@@ -228,11 +228,36 @@ Do not hard code values inside the DAG and then change them manually according t
You can use environment variables to parameterize the DAG.
-.. code-block::
+.. code-block:: python
+
+ import os
+
+ dest = os.environ.get(
+ "MY_DAG_DEST_PATH",
+ "s3://default-target/path/"
+ )
+
+Mocking variables and connections
+=================================
+
Review comment:
I think it would be great to add a little context here. In which situations you would like to do that? I am not 100% sure here if we are talking about "user facing' mocking or "contributors" mocking. If this is user-facing, then it would be great to explain when mocking might be useful. If this is "contributors-facing" then it should be moved to CONTRIBUTORS.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #13502: Add docs about mocking variables and connections
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #13502:
URL: https://github.com/apache/airflow/pull/13502#discussion_r552504057
##########
File path: docs/apache-airflow/best-practices.rst
##########
@@ -228,11 +228,36 @@ Do not hard code values inside the DAG and then change them manually according t
You can use environment variables to parameterize the DAG.
-.. code-block::
+.. code-block:: python
+
+ import os
+
+ dest = os.environ.get(
+ "MY_DAG_DEST_PATH",
+ "s3://default-target/path/"
+ )
+
+Mocking variables and connections
+=================================
+
Review comment:
I think this is useful for the end user. I've seen similar questions on Slack many times, including once yesterday. See: https://apache-airflow.slack.com/archives/CSS36QQS1/p1609874991010700
I have added some context in which cases, this can be helpful.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #13502: Add docs about mocking variables and connections
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #13502:
URL: https://github.com/apache/airflow/pull/13502#discussion_r552503227
##########
File path: docs/apache-airflow/best-practices.rst
##########
@@ -228,11 +228,36 @@ Do not hard code values inside the DAG and then change them manually according t
You can use environment variables to parameterize the DAG.
-.. code-block::
+.. code-block:: python
+
+ import os
+
+ dest = os.environ.get(
+ "MY_DAG_DEST_PATH",
+ "s3://default-target/path/"
+ )
+
+Mocking variables and connections
+=================================
+
+You can set an environment variable to configure variables or connections to limit interaction with the database.
Review comment:
```suggestion
When you write tests for code that uses variables or a connection, you must ensure that they exist when you run the tests. The obvious solution is to save these objects to the database so they can be read while your code is executing. However, reading and writing objects to the database are burdened with additional time overhead. In order to speed up the test execution, it is worth simulating the existence of these objects without saving them to the database. For this, you can create environment variables with mocking :any:`os.environ` using :meth:`unittest.mock.patch.dict`.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #13502: Add docs about mocking variables and connections
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #13502:
URL: https://github.com/apache/airflow/pull/13502#discussion_r552357176
##########
File path: docs/apache-airflow/best-practices.rst
##########
@@ -90,13 +90,13 @@ This can result in a lot of open connections.
The best way of using variables is via a Jinja template, which will delay reading the value until the task execution. The template syntax to do this is:
-.. code-block::
+.. code-block:: jinja
{{ var.value.<variable_name> }}
or if you need to deserialize a json object from the variable :
-.. code-block::
+.. code-block:: jinja
Review comment:
```suggestion
.. code-block::
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #13502: Add docs about mocking variables and connections
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #13502:
URL: https://github.com/apache/airflow/pull/13502
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #13502: Add docs about mocking variables and connections
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #13502:
URL: https://github.com/apache/airflow/pull/13502#discussion_r552357125
##########
File path: docs/apache-airflow/best-practices.rst
##########
@@ -90,13 +90,13 @@ This can result in a lot of open connections.
The best way of using variables is via a Jinja template, which will delay reading the value until the task execution. The template syntax to do this is:
-.. code-block::
+.. code-block:: jinja
Review comment:
```suggestion
.. code-block::
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org