You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/06 07:24:38 UTC
[GitHub] [airflow] mhenc commented on a change in pull request #10172: Update guide for Google Cloud Secret Manager Backend
mhenc commented on a change in pull request #10172:
URL: https://github.com/apache/airflow/pull/10172#discussion_r466187724
##########
File path: docs/howto/use-alternative-secrets-backend.rst
##########
@@ -433,8 +460,43 @@ When ``gcp_key_path`` is not provided, it will use the Application Default Crede
* `google.auth.default <https://google-auth.readthedocs.io/en/latest/reference/google.auth.html#google.auth.default>`__
* `Setting Up Authentication for Server to Server Production Applications <https://cloud.google.com/docs/authentication/production>`__
-The value of the Secrets Manager secret id must be the :ref:`connection URI representation <generating_connection_uri>`
-of the connection object.
+Managing a secrets
Review comment:
remove 'a'
##########
File path: airflow/providers/google/cloud/secrets/secret_manager.py
##########
@@ -64,9 +64,9 @@ class CloudSecretManagerBackend(BaseSecretsBackend, LoggingMixin):
:type gcp_keyfile_dict: dict
:param gcp_scopes: Comma-separated string containing GCP scopes
:type gcp_scopes: str
- :param project_id: Project id (if you want to override the project_id from credentials)
+ :param project_id: Project ID. If not passed, the project ID from credentials will be used.
Review comment:
Maybe add something what is the values used for like
"Project Id to read the secrets from. If not provided, thethe project ID from credentials id used"
##########
File path: docs/howto/use-alternative-secrets-backend.rst
##########
@@ -383,48 +383,75 @@ Note that the secret ``Key`` is ``value``, and secret ``Value`` is ``world`` and
.. _secret_manager_backend:
-GCP Secret Manager Backend
-^^^^^^^^^^^^^^^^^^^^^^^^^^
+Google Cloud Secret Manager Backend
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-To enable GCP Secrets Manager to retrieve connection/variables, specify :py:class:`~airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend`
-as the ``backend`` in ``[secrets]`` section of ``airflow.cfg``.
+This topic describes how to configure Airflow to use `Secret Manager <https://cloud.google.com/secret-manager/docs>`__ as
+a secret bakcned and how to manage secrets.
-Available parameters to ``backend_kwargs``:
+Before you begin
+""""""""""""""""
-* ``connections_prefix``: Specifies the prefix of the secret to read to get Connections.
-* ``variables_prefix``: Specifies the prefix of the secret to read to get Variables.
-* ``gcp_key_path``: Path to GCP Credential JSON file
-* ``gcp_scopes``: Comma-separated string containing GCP scopes
-* ``sep``: separator used to concatenate connections_prefix and conn_id. Default: "-"
+`Configure Secret Manager and your local environment <https://cloud.google.com/secret-manager/docs/configuring-secret-manager>`__, once per project.
-Note: The full GCP Secrets Manager secret id should follow the pattern "[a-zA-Z0-9-_]".
+Enabling the secret backend
+"""""""""""""""""""""""""""
-Here is a sample configuration if you want to just retrieve connections:
+To enable the secret backend for Google Cloud Secrets Manager to retrieve connection/variables,
+specify :py:class:`~airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend`
+as the ``backend`` in ``[secrets]`` section of ``airflow.cfg``.
+
+Here is a sample configuration if you want to use it:
.. code-block:: ini
[secrets]
backend = airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
- backend_kwargs = {"connections_prefix": "airflow-connections", "sep": "-"}
-Here is a sample configuration if you want to just retrieve variables:
+You can also set this with environment variables.
-.. code-block:: ini
+.. code-block:: bash
- [secrets]
- backend = airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
- backend_kwargs = {"variables_prefix": "airflow-variables", "sep": "-"}
+ export AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
-and if you want to retrieve both Variables and connections use the following sample config:
+You can verify the correct setting of the configuration options with the ``airflw config get-value`` command.
+
+.. code-block:: bash
+
+ $ airflow config get-value secrets backend
+ airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
+
+Additionals options
+"""""""""""""""""""
+
+The next step is to configure additional configuration options using the ``backend_kwargs`` options.
+
+* ``connections_prefix``: Specifies the prefix of the secret to read to get Connections. Default: ``"airflow-connections"``
+* ``variables_prefix``: Specifies the prefix of the secret to read to get Variables. Default: ``"airflow-variables"``
+* ``gcp_key_path``: Path to GCP Credential JSON file.
+* ``gcp_keyfile_dict``: Dictionary of keyfile parameters.
+* ``gcp_scopes``: Comma-separated string containing GCP scopes.
+* ``sep``: Separator used to concatenate connections_prefix and conn_id. Default: "-"
+* ``project_id``: Project ID. If not passed, the project ID from credentials will be used.
+
+All options should be passed as a JSON dictionary.
+
+For example, if you want to set parameter ``connections_prefix`` to ``"airflow-tenant-primary"`` and parameter ``variables_prefix`` to ``"variables_prefix"``, your configuration file should look like this:
.. code-block:: ini
[secrets]
backend = airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
- backend_kwargs = {"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "sep": "-"}
+ backend_kwargs = {"connections_prefix": "airflow-tenant-primary", "variables_prefix": "airflow-tenant-primary"}
+
+Set-up credentials
+""""""""""""""""""
+You can configure the credentiaps in three ways:
Review comment:
typo : credentiaps
##########
File path: docs/howto/use-alternative-secrets-backend.rst
##########
@@ -383,48 +383,75 @@ Note that the secret ``Key`` is ``value``, and secret ``Value`` is ``world`` and
.. _secret_manager_backend:
-GCP Secret Manager Backend
-^^^^^^^^^^^^^^^^^^^^^^^^^^
+Google Cloud Secret Manager Backend
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-To enable GCP Secrets Manager to retrieve connection/variables, specify :py:class:`~airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend`
-as the ``backend`` in ``[secrets]`` section of ``airflow.cfg``.
+This topic describes how to configure Airflow to use `Secret Manager <https://cloud.google.com/secret-manager/docs>`__ as
+a secret bakcned and how to manage secrets.
Review comment:
typo: bakcned
##########
File path: docs/howto/use-alternative-secrets-backend.rst
##########
@@ -383,48 +383,75 @@ Note that the secret ``Key`` is ``value``, and secret ``Value`` is ``world`` and
.. _secret_manager_backend:
-GCP Secret Manager Backend
-^^^^^^^^^^^^^^^^^^^^^^^^^^
+Google Cloud Secret Manager Backend
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-To enable GCP Secrets Manager to retrieve connection/variables, specify :py:class:`~airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend`
-as the ``backend`` in ``[secrets]`` section of ``airflow.cfg``.
+This topic describes how to configure Airflow to use `Secret Manager <https://cloud.google.com/secret-manager/docs>`__ as
+a secret bakcned and how to manage secrets.
-Available parameters to ``backend_kwargs``:
+Before you begin
+""""""""""""""""
-* ``connections_prefix``: Specifies the prefix of the secret to read to get Connections.
-* ``variables_prefix``: Specifies the prefix of the secret to read to get Variables.
-* ``gcp_key_path``: Path to GCP Credential JSON file
-* ``gcp_scopes``: Comma-separated string containing GCP scopes
-* ``sep``: separator used to concatenate connections_prefix and conn_id. Default: "-"
+`Configure Secret Manager and your local environment <https://cloud.google.com/secret-manager/docs/configuring-secret-manager>`__, once per project.
-Note: The full GCP Secrets Manager secret id should follow the pattern "[a-zA-Z0-9-_]".
+Enabling the secret backend
+"""""""""""""""""""""""""""
-Here is a sample configuration if you want to just retrieve connections:
+To enable the secret backend for Google Cloud Secrets Manager to retrieve connection/variables,
+specify :py:class:`~airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend`
Review comment:
This requires installing backport-operators. Maybe it's worth to mention that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org