You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/05 14:49:25 UTC
[GitHub] [airflow] mik-laj opened a new pull request #10172: Update guide for Google Cloud Secret Manager Backend
mik-laj opened a new pull request #10172:
URL: https://github.com/apache/airflow/pull/10172
Based on the feedback from users, I decided to write a few more sections.
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #10172: Update guide for Google Cloud Secret Manager Backend
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #10172:
URL: https://github.com/apache/airflow/pull/10172#discussion_r466536026
##########
File path: docs/howto/use-alternative-secrets-backend.rst
##########
@@ -383,48 +383,75 @@ Note that the secret ``Key`` is ``value``, and secret ``Value`` is ``world`` and
.. _secret_manager_backend:
-GCP Secret Manager Backend
-^^^^^^^^^^^^^^^^^^^^^^^^^^
+Google Cloud Secret Manager Backend
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-To enable GCP Secrets Manager to retrieve connection/variables, specify :py:class:`~airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend`
-as the ``backend`` in ``[secrets]`` section of ``airflow.cfg``.
+This topic describes how to configure Airflow to use `Secret Manager <https://cloud.google.com/secret-manager/docs>`__ as
+a secret bakcned and how to manage secrets.
-Available parameters to ``backend_kwargs``:
+Before you begin
+""""""""""""""""
-* ``connections_prefix``: Specifies the prefix of the secret to read to get Connections.
-* ``variables_prefix``: Specifies the prefix of the secret to read to get Variables.
-* ``gcp_key_path``: Path to GCP Credential JSON file
-* ``gcp_scopes``: Comma-separated string containing GCP scopes
-* ``sep``: separator used to concatenate connections_prefix and conn_id. Default: "-"
+`Configure Secret Manager and your local environment <https://cloud.google.com/secret-manager/docs/configuring-secret-manager>`__, once per project.
-Note: The full GCP Secrets Manager secret id should follow the pattern "[a-zA-Z0-9-_]".
+Enabling the secret backend
+"""""""""""""""""""""""""""
-Here is a sample configuration if you want to just retrieve connections:
+To enable the secret backend for Google Cloud Secrets Manager to retrieve connection/variables,
+specify :py:class:`~airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend`
Review comment:
Backport Packages have their own documentation - - and I already have a mechanism to incorpoare some extra information in it - I will extract some of the useful GCP guides ther with the next wave of backport packages.
The documentation is here: https://github.com/apache/airflow/tree/master/airflow/providers/google
And when released it can be found in PyPI https://pypi.org/project/apache-airflow-backport-providers-google/2020.6.24/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj merged pull request #10172: Update guide for Google Cloud Secret Manager Backend
Posted by GitBox <gi...@apache.org>.
mik-laj merged pull request #10172:
URL: https://github.com/apache/airflow/pull/10172
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #10172: Update guide for Google Cloud Secret Manager Backend
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #10172:
URL: https://github.com/apache/airflow/pull/10172#issuecomment-670022730
@potiuk I took into account all the comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10172: Update guide for Google Cloud Secret Manager Backend
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10172:
URL: https://github.com/apache/airflow/pull/10172#discussion_r466328531
##########
File path: docs/howto/use-alternative-secrets-backend.rst
##########
@@ -383,48 +383,75 @@ Note that the secret ``Key`` is ``value``, and secret ``Value`` is ``world`` and
.. _secret_manager_backend:
-GCP Secret Manager Backend
-^^^^^^^^^^^^^^^^^^^^^^^^^^
+Google Cloud Secret Manager Backend
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-To enable GCP Secrets Manager to retrieve connection/variables, specify :py:class:`~airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend`
-as the ``backend`` in ``[secrets]`` section of ``airflow.cfg``.
+This topic describes how to configure Airflow to use `Secret Manager <https://cloud.google.com/secret-manager/docs>`__ as
+a secret bakcned and how to manage secrets.
-Available parameters to ``backend_kwargs``:
+Before you begin
+""""""""""""""""
-* ``connections_prefix``: Specifies the prefix of the secret to read to get Connections.
-* ``variables_prefix``: Specifies the prefix of the secret to read to get Variables.
-* ``gcp_key_path``: Path to GCP Credential JSON file
-* ``gcp_scopes``: Comma-separated string containing GCP scopes
-* ``sep``: separator used to concatenate connections_prefix and conn_id. Default: "-"
+`Configure Secret Manager and your local environment <https://cloud.google.com/secret-manager/docs/configuring-secret-manager>`__, once per project.
-Note: The full GCP Secrets Manager secret id should follow the pattern "[a-zA-Z0-9-_]".
+Enabling the secret backend
+"""""""""""""""""""""""""""
-Here is a sample configuration if you want to just retrieve connections:
+To enable the secret backend for Google Cloud Secrets Manager to retrieve connection/variables,
+specify :py:class:`~airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend`
Review comment:
Unfortunately, this is documentation for Airflow 2.0 where these packages don't work. We don't have documentation for Airflow 1.10 with backport packages.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mhenc commented on a change in pull request #10172: Update guide for Google Cloud Secret Manager Backend
Posted by GitBox <gi...@apache.org>.
mhenc commented on a change in pull request #10172:
URL: https://github.com/apache/airflow/pull/10172#discussion_r466187724
##########
File path: docs/howto/use-alternative-secrets-backend.rst
##########
@@ -433,8 +460,43 @@ When ``gcp_key_path`` is not provided, it will use the Application Default Crede
* `google.auth.default <https://google-auth.readthedocs.io/en/latest/reference/google.auth.html#google.auth.default>`__
* `Setting Up Authentication for Server to Server Production Applications <https://cloud.google.com/docs/authentication/production>`__
-The value of the Secrets Manager secret id must be the :ref:`connection URI representation <generating_connection_uri>`
-of the connection object.
+Managing a secrets
Review comment:
remove 'a'
##########
File path: airflow/providers/google/cloud/secrets/secret_manager.py
##########
@@ -64,9 +64,9 @@ class CloudSecretManagerBackend(BaseSecretsBackend, LoggingMixin):
:type gcp_keyfile_dict: dict
:param gcp_scopes: Comma-separated string containing GCP scopes
:type gcp_scopes: str
- :param project_id: Project id (if you want to override the project_id from credentials)
+ :param project_id: Project ID. If not passed, the project ID from credentials will be used.
Review comment:
Maybe add something what is the values used for like
"Project Id to read the secrets from. If not provided, thethe project ID from credentials id used"
##########
File path: docs/howto/use-alternative-secrets-backend.rst
##########
@@ -383,48 +383,75 @@ Note that the secret ``Key`` is ``value``, and secret ``Value`` is ``world`` and
.. _secret_manager_backend:
-GCP Secret Manager Backend
-^^^^^^^^^^^^^^^^^^^^^^^^^^
+Google Cloud Secret Manager Backend
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-To enable GCP Secrets Manager to retrieve connection/variables, specify :py:class:`~airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend`
-as the ``backend`` in ``[secrets]`` section of ``airflow.cfg``.
+This topic describes how to configure Airflow to use `Secret Manager <https://cloud.google.com/secret-manager/docs>`__ as
+a secret bakcned and how to manage secrets.
-Available parameters to ``backend_kwargs``:
+Before you begin
+""""""""""""""""
-* ``connections_prefix``: Specifies the prefix of the secret to read to get Connections.
-* ``variables_prefix``: Specifies the prefix of the secret to read to get Variables.
-* ``gcp_key_path``: Path to GCP Credential JSON file
-* ``gcp_scopes``: Comma-separated string containing GCP scopes
-* ``sep``: separator used to concatenate connections_prefix and conn_id. Default: "-"
+`Configure Secret Manager and your local environment <https://cloud.google.com/secret-manager/docs/configuring-secret-manager>`__, once per project.
-Note: The full GCP Secrets Manager secret id should follow the pattern "[a-zA-Z0-9-_]".
+Enabling the secret backend
+"""""""""""""""""""""""""""
-Here is a sample configuration if you want to just retrieve connections:
+To enable the secret backend for Google Cloud Secrets Manager to retrieve connection/variables,
+specify :py:class:`~airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend`
+as the ``backend`` in ``[secrets]`` section of ``airflow.cfg``.
+
+Here is a sample configuration if you want to use it:
.. code-block:: ini
[secrets]
backend = airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
- backend_kwargs = {"connections_prefix": "airflow-connections", "sep": "-"}
-Here is a sample configuration if you want to just retrieve variables:
+You can also set this with environment variables.
-.. code-block:: ini
+.. code-block:: bash
- [secrets]
- backend = airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
- backend_kwargs = {"variables_prefix": "airflow-variables", "sep": "-"}
+ export AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
-and if you want to retrieve both Variables and connections use the following sample config:
+You can verify the correct setting of the configuration options with the ``airflw config get-value`` command.
+
+.. code-block:: bash
+
+ $ airflow config get-value secrets backend
+ airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
+
+Additionals options
+"""""""""""""""""""
+
+The next step is to configure additional configuration options using the ``backend_kwargs`` options.
+
+* ``connections_prefix``: Specifies the prefix of the secret to read to get Connections. Default: ``"airflow-connections"``
+* ``variables_prefix``: Specifies the prefix of the secret to read to get Variables. Default: ``"airflow-variables"``
+* ``gcp_key_path``: Path to GCP Credential JSON file.
+* ``gcp_keyfile_dict``: Dictionary of keyfile parameters.
+* ``gcp_scopes``: Comma-separated string containing GCP scopes.
+* ``sep``: Separator used to concatenate connections_prefix and conn_id. Default: "-"
+* ``project_id``: Project ID. If not passed, the project ID from credentials will be used.
+
+All options should be passed as a JSON dictionary.
+
+For example, if you want to set parameter ``connections_prefix`` to ``"airflow-tenant-primary"`` and parameter ``variables_prefix`` to ``"variables_prefix"``, your configuration file should look like this:
.. code-block:: ini
[secrets]
backend = airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
- backend_kwargs = {"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "sep": "-"}
+ backend_kwargs = {"connections_prefix": "airflow-tenant-primary", "variables_prefix": "airflow-tenant-primary"}
+
+Set-up credentials
+""""""""""""""""""
+You can configure the credentiaps in three ways:
Review comment:
typo : credentiaps
##########
File path: docs/howto/use-alternative-secrets-backend.rst
##########
@@ -383,48 +383,75 @@ Note that the secret ``Key`` is ``value``, and secret ``Value`` is ``world`` and
.. _secret_manager_backend:
-GCP Secret Manager Backend
-^^^^^^^^^^^^^^^^^^^^^^^^^^
+Google Cloud Secret Manager Backend
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-To enable GCP Secrets Manager to retrieve connection/variables, specify :py:class:`~airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend`
-as the ``backend`` in ``[secrets]`` section of ``airflow.cfg``.
+This topic describes how to configure Airflow to use `Secret Manager <https://cloud.google.com/secret-manager/docs>`__ as
+a secret bakcned and how to manage secrets.
Review comment:
typo: bakcned
##########
File path: docs/howto/use-alternative-secrets-backend.rst
##########
@@ -383,48 +383,75 @@ Note that the secret ``Key`` is ``value``, and secret ``Value`` is ``world`` and
.. _secret_manager_backend:
-GCP Secret Manager Backend
-^^^^^^^^^^^^^^^^^^^^^^^^^^
+Google Cloud Secret Manager Backend
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-To enable GCP Secrets Manager to retrieve connection/variables, specify :py:class:`~airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend`
-as the ``backend`` in ``[secrets]`` section of ``airflow.cfg``.
+This topic describes how to configure Airflow to use `Secret Manager <https://cloud.google.com/secret-manager/docs>`__ as
+a secret bakcned and how to manage secrets.
-Available parameters to ``backend_kwargs``:
+Before you begin
+""""""""""""""""
-* ``connections_prefix``: Specifies the prefix of the secret to read to get Connections.
-* ``variables_prefix``: Specifies the prefix of the secret to read to get Variables.
-* ``gcp_key_path``: Path to GCP Credential JSON file
-* ``gcp_scopes``: Comma-separated string containing GCP scopes
-* ``sep``: separator used to concatenate connections_prefix and conn_id. Default: "-"
+`Configure Secret Manager and your local environment <https://cloud.google.com/secret-manager/docs/configuring-secret-manager>`__, once per project.
-Note: The full GCP Secrets Manager secret id should follow the pattern "[a-zA-Z0-9-_]".
+Enabling the secret backend
+"""""""""""""""""""""""""""
-Here is a sample configuration if you want to just retrieve connections:
+To enable the secret backend for Google Cloud Secrets Manager to retrieve connection/variables,
+specify :py:class:`~airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend`
Review comment:
This requires installing backport-operators. Maybe it's worth to mention that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org