You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/08 09:14:35 UTC
[GitHub] [airflow] mik-laj opened a new pull request #10241: Create "Managing variable" in howto directory
mik-laj opened a new pull request #10241:
URL: https://github.com/apache/airflow/pull/10241
I will soon want to develop this feature more and need some space to describe improvements.
https://github.com/apache/airflow/issues/9856
https://github.com/apache/airflow/issues/9855
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj merged pull request #10241: Create "Managing variable" in howto directory
Posted by GitBox <gi...@apache.org>.
mik-laj merged pull request #10241:
URL: https://github.com/apache/airflow/pull/10241
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #10241: Create "Managing variable" in howto directory
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #10241:
URL: https://github.com/apache/airflow/pull/10241#issuecomment-671053835
Build docs is green.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10241: Create "Managing variable" in howto directory
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10241:
URL: https://github.com/apache/airflow/pull/10241#discussion_r467550010
##########
File path: docs/howto/variable.rst
##########
@@ -0,0 +1,72 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Managing Connections
+====================
+
+Variables are a generic way to store and retrieve arbitrary content or
+settings as a simple key value store within Airflow. Variables can be
+listed, created, updated and deleted from the UI (``Admin -> Variables``),
+code or CLI.
+
+.. image:: ../img/variable_hidden.png
+
+See the :ref:`Variables Concepts <concepts-variables>` documentation for
+more information.
+
+Storing Variables in Environment Variables
Review comment:
Environment variables are helpful because they can be set very easily in a containerized environment and their contents can be determined during deployment.
https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/
It could also contain secrets.
https://kubernetes.io/docs/concepts/configuration/secret/
In the DAG file, you can also read environment variables, but the use of Airflow variables allows for their easier configuration in the development environment, e.g. access to UI, easier development - access to variable in Jinja, and thanks to setting Airflow variables using environment variables also easy deployment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #10241: Create "Managing variable" in howto directory
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #10241:
URL: https://github.com/apache/airflow/pull/10241#discussion_r467508200
##########
File path: docs/howto/variable.rst
##########
@@ -15,7 +15,7 @@
specific language governing permissions and limitations
under the License.
-Managing Connections
+Managing Variables
====================
Review comment:
Length of the underline (====) is wrong
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JeffryMAC commented on a change in pull request #10241: Create "Managing variable" in howto directory
Posted by GitBox <gi...@apache.org>.
JeffryMAC commented on a change in pull request #10241:
URL: https://github.com/apache/airflow/pull/10241#discussion_r467501205
##########
File path: docs/howto/variable.rst
##########
@@ -0,0 +1,72 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Managing Connections
+====================
+
+Variables are a generic way to store and retrieve arbitrary content or
+settings as a simple key value store within Airflow. Variables can be
+listed, created, updated and deleted from the UI (``Admin -> Variables``),
+code or CLI.
+
+.. image:: ../img/variable_hidden.png
+
+See the :ref:`Variables Concepts <concepts-variables>` documentation for
+more information.
+
+Storing Variables in Environment Variables
Review comment:
If I may add... there the previous paragraph explain what is Variable.
This paragraph explain how to store variable as Environment Variables (This is a **How to** but not **What is**)
It's not clear to me why to do that. What this functionality gives and when someone would need to use it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10241: Create "Managing variable" in howto directory
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10241:
URL: https://github.com/apache/airflow/pull/10241#discussion_r467583127
##########
File path: docs/howto/variable.rst
##########
@@ -0,0 +1,72 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Managing Variables
+==================
+
+Variables are a generic way to store and retrieve arbitrary content or
+settings as a simple key value store within Airflow. Variables can be
+listed, created, updated and deleted from the UI (``Admin -> Variables``),
+code or CLI.
+
+.. image:: ../img/variable_hidden.png
+
+See the :ref:`Variables Concepts <concepts-variables>` documentation for
Review comment:
```suggestion
See the :ref:`Variables Concepts <concepts:variables>` documentation for
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #10241: Create "Managing variable" in howto directory
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #10241:
URL: https://github.com/apache/airflow/pull/10241#discussion_r467560888
##########
File path: docs/howto/variable.rst
##########
@@ -0,0 +1,72 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Managing Variables
+==================
+
+Variables are a generic way to store and retrieve arbitrary content or
+settings as a simple key value store within Airflow. Variables can be
+listed, created, updated and deleted from the UI (``Admin -> Variables``),
+code or CLI.
+
+.. image:: ../img/variable_hidden.png
+
+See the :ref:`Variables Concepts <concepts-variables>` documentation for
+more information.
+
+Storing Variables in Environment Variables
+------------------------------------------
+
+.. versionadded:: 1.10.10
+
+Airflow Variables can also be created and managed using Environment Variables. The environment variable
+naming convention is :envvar:`AIRFLOW_VAR_{VARIABLE_NAME}`, all uppercase.
+So if your variable key is ``FOO`` then the variable name should be ``AIRFLOW_VAR_FOO``.
+
+For example,
+
+.. code-block:: bash
+
+ export AIRFLOW_VAR_FOO=BAR
+
+ # To use JSON, store them as JSON strings
+ export AIRFLOW_VAR_FOO_BAZ='{"hello":"world"}'
+
+You can use them in your DAGs as:
+
+.. code-block:: python
+
+ from airflow.models import Variable
+ foo = Variable.get("foo")
+ foo_json = Variable.get("foo_baz", deserialize_json=True)
+
+.. note::
+
+ Single underscores surround ``VAR``. This is in contrast with the way ``airflow.cfg``
+ parameters are stored, where double underscores surround the config section name.
+ Variables set using Environment Variables would not appear in the Airflow UI but you will
+ be able to use it in your DAG file.
+
+Securing Variables
+------------------
+
+Airflow uses `Fernet <https://github.com/fernet/spec/>`__ to encrypt variables stored the metastore database.
+It guarantees that without the encryption password, content cannot be manipulated or read
+without the key. For information on configuring Fernet, look at :ref:`security/fernet`.
+
+In addition to retrieving variables from environment variables or the metastore database, you can enable
+an secrets backend to retrieve variablles. For more details see :doc:`../secrets-backend/index`
Review comment:
```suggestion
Securing Variables
------------------
Airflow uses `Fernet <https://github.com/fernet/spec/>`__ to encrypt variables stored in the metastore database.
It guarantees that without the encryption password, content cannot be manipulated or read
without the key. For information on configuring Fernet, look at :ref:`security/fernet`.
In addition to retrieving variables from environment variables or the metastore database, you can enable
a secrets backend to retrieve variables. For more details see :doc:`../secrets-backend/index`
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10241: Create "Managing variable" in howto directory
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10241:
URL: https://github.com/apache/airflow/pull/10241#discussion_r467583171
##########
File path: docs/howto/variable.rst
##########
@@ -0,0 +1,72 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Managing Variables
+==================
+
+Variables are a generic way to store and retrieve arbitrary content or
+settings as a simple key value store within Airflow. Variables can be
+listed, created, updated and deleted from the UI (``Admin -> Variables``),
+code or CLI.
+
+.. image:: ../img/variable_hidden.png
+
+See the :ref:`Variables Concepts <concepts-variables>` documentation for
+more information.
+
+Storing Variables in Environment Variables
+------------------------------------------
+
+.. versionadded:: 1.10.10
+
+Airflow Variables can also be created and managed using Environment Variables. The environment variable
+naming convention is :envvar:`AIRFLOW_VAR_{VARIABLE_NAME}`, all uppercase.
+So if your variable key is ``FOO`` then the variable name should be ``AIRFLOW_VAR_FOO``.
+
+For example,
+
+.. code-block:: bash
+
+ export AIRFLOW_VAR_FOO=BAR
+
+ # To use JSON, store them as JSON strings
+ export AIRFLOW_VAR_FOO_BAZ='{"hello":"world"}'
+
+You can use them in your DAGs as:
+
+.. code-block:: python
+
+ from airflow.models import Variable
+ foo = Variable.get("foo")
+ foo_json = Variable.get("foo_baz", deserialize_json=True)
+
+.. note::
+
+ Single underscores surround ``VAR``. This is in contrast with the way ``airflow.cfg``
+ parameters are stored, where double underscores surround the config section name.
+ Variables set using Environment Variables would not appear in the Airflow UI but you will
+ be able to use it in your DAG file.
+
+Securing Variables
+------------------
+
+Airflow uses `Fernet <https://github.com/fernet/spec/>`__ to encrypt variables stored in the metastore database.
+It guarantees that without the encryption password, content cannot be manipulated or read
+without the key. For information on configuring Fernet, look at :ref:`security/fernet`.
+
+In addition to retrieving variables from environment variables or the metastore database, you can enable
+a secrets backend to retrieve variables. For more details see :doc:`../secrets-backend/index`
Review comment:
```suggestion
a secrets backend to retrieve variables. For more details see :doc:`secrets-backend/index`
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JeffryMAC commented on a change in pull request #10241: Create "Managing variable" in howto directory
Posted by GitBox <gi...@apache.org>.
JeffryMAC commented on a change in pull request #10241:
URL: https://github.com/apache/airflow/pull/10241#discussion_r467544808
##########
File path: docs/howto/variable.rst
##########
@@ -0,0 +1,72 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Managing Connections
+====================
+
+Variables are a generic way to store and retrieve arbitrary content or
+settings as a simple key value store within Airflow. Variables can be
+listed, created, updated and deleted from the UI (``Admin -> Variables``),
+code or CLI.
+
+.. image:: ../img/variable_hidden.png
+
+See the :ref:`Variables Concepts <concepts-variables>` documentation for
+more information.
+
+Storing Variables in Environment Variables
Review comment:
I was pointing that because I don't know the answers and there is no where in the docs where this info available.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10241: Create "Managing variable" in howto directory
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10241:
URL: https://github.com/apache/airflow/pull/10241#discussion_r467550086
##########
File path: docs/howto/variable.rst
##########
@@ -0,0 +1,72 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Managing Variables
+====================
Review comment:
```suggestion
Managing Variables
==================
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10241: Create "Managing variable" in howto directory
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10241:
URL: https://github.com/apache/airflow/pull/10241#discussion_r467503305
##########
File path: docs/howto/variable.rst
##########
@@ -0,0 +1,72 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Managing Connections
+====================
+
+Variables are a generic way to store and retrieve arbitrary content or
+settings as a simple key value store within Airflow. Variables can be
+listed, created, updated and deleted from the UI (``Admin -> Variables``),
+code or CLI.
+
+.. image:: ../img/variable_hidden.png
+
+See the :ref:`Variables Concepts <concepts-variables>` documentation for
+more information.
+
+Storing Variables in Environment Variables
Review comment:
Can you add it in a separate PR? I just want to create a page in this PR based on the existing information. I would have added sections, but we should add similar information to other pages too, and changes to other pages are beyond the scope of this PR. WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #10241: Create "Managing variable" in howto directory
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #10241:
URL: https://github.com/apache/airflow/pull/10241#discussion_r467455156
##########
File path: docs/howto/variable.rst
##########
@@ -0,0 +1,72 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Managing Connections
Review comment:
```suggestion
Managing Variables
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org