You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/07/03 10:38:37 UTC
[GitHub] [airflow] elwinarens opened a new pull request #9639: Add secrets backend for microsoft azure key vault
elwinarens opened a new pull request #9639:
URL: https://github.com/apache/airflow/pull/9639
Add secrets backend for microsoft azure key vault. Related issue https://github.com/apache/airflow/issues/8258.
---
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Target Github ISSUE in description if exists
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#discussion_r452664124
##########
File path: airflow/providers/microsoft/azure/secrets/azure_key_vault.py
##########
@@ -0,0 +1,131 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Optional
+
+from azure.core.exceptions import ResourceNotFoundError
+from azure.identity import DefaultAzureCredential
+from azure.keyvault.secrets import SecretClient
+from cached_property import cached_property
+
+from airflow.secrets import BaseSecretsBackend
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class AzureKeyVaultBackend(BaseSecretsBackend, LoggingMixin):
+ """
+ Retrieves Airflow Connections or Variables from Azure Key Vault secrets.
+
+ The Azure Key Vault can be configred as a secrets backend in the ``airflow.cfg``:
+
+ .. code-block:: ini
+
+ [secrets]
+ backend = airflow.providers.microsoft.azure.secrets.azure_key_vault.AzureKeyVaultBackend
+ backend_kwargs = {"connections_prefix": "airflow-connections", "vault_url": "<azure_key_vault_uri>"}
+
+ For example, if the secrets prefix is ``airflow-connections-smtp-default``, this would be accessible
+ if you provide ``{"connections_prefix": "airflow-connections"}`` and request conn_id ``smtp-default``.
+ And if variables prefix is ``airflow-variables-hello``, this would be accessible
+ if you provide ``{"variables_prefix": "airflow-variables"}`` and request variable key ``hello``.
+
+ :param vault_url: The URL of an Azure Key Vault to use
+ :type vault_url: str
+ :param connections_prefix: Specifies the prefix of the secret to read to get Connections
+ :type connections_prefix: str
+ :param variables_prefix: Specifies the prefix of the secret to read to get Variables
+ :type variables_prefix: str
+ :param sep: separator used to concatenate secret_prefix and secret_id. Default: "-"
+ :type sep: str
+ """
+
+ def __init__(
+ self,
+ connections_prefix: str = 'airflow-connections',
+ variables_prefix: str = 'airflow-variables',
+ vault_url: str = '',
+ sep: str = '-',
+ **kwargs
+ ):
+ super().__init__()
+ self.vault_url = vault_url
+ self.connections_prefix = connections_prefix.rstrip(sep)
+ self.variables_prefix = variables_prefix.rstrip(sep)
+ self.sep = sep
+ self.kwargs = kwargs
+
+ @cached_property
+ def client(self):
+ """
+ Create a Azure Key Vault client.
+ """
+ credential = DefaultAzureCredential()
+ client = SecretClient(
+ vault_url=self.vault_url,
+ credential=credential,
+ **self.kwargs
+ )
+ return client
+
+ def get_conn_uri(self, conn_id: str) -> Optional[str]:
+ """
+ Get an Airflow Connection URI from an Azure Key Vault secret
+
+ :param conn_id: The Airflow connection id to retrieve
+ :type conn_id: str
+ """
+ name = self.build_path(self.connections_prefix, conn_id, self.sep)
+ return self.get_secret_value(name)
+
+ def get_variable(self, key: str) -> Optional[str]:
+ """
+ Get an Airflow Variable from an Azure Key Vault secret.
+
+ :param key: Variable Key
+ :type key: str
+ :return: Variable Value
+ """
+ name = self.build_path(self.variables_prefix, key, self.sep)
+ return self.get_secret_value(name)
+
+ @staticmethod
+ def build_path(path_prefix: str, secret_id: str, sep: str = '-') -> str:
+ """
+ Given a path_prefix and secret_id, build a valid secret name for the Azure Key Vault Backend.
+ Also replaces underscore in the path with dashes to support easy switching between
+ environmentvariables, so ``connection_default`` becomes ``connection-default``.
+
+ :param path_prefix: The path prefix of the secret to retrieve
+ :type path_prefix: str
+ :param secret_id: Name of the secret
+ :type secret_id: str
+ :param sep: Separator used to concatenate path_prefix and secret_id
+ :type sep: str
+ """
+ path = f'{path_prefix}{sep}{secret_id}'
+ return path.replace('_', sep)
+
+ def get_secret_value(self, name: str) -> Optional[str]:
Review comment:
We will have hooks for that, for example: https://github.com/apache/airflow/blob/master/airflow/providers/hashicorp/hooks/vault.py
WDYT, would that be useful for your use-case?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#discussion_r450807848
##########
File path: airflow/providers/microsoft/azure/secrets/azure_key_vault.py
##########
@@ -0,0 +1,107 @@
+"""
Review comment:
Also in airflow/providers/microsoft/azure/secrets/__init__.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-681107887
@flolas Please feel free to create a new PR, looks like @elwinarens might be busy
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens commented on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens commented on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-690866681
@kaxil that option is not available, because this PR is not under a personal user. Feel free to put up a new PR and close this one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens edited a comment on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens edited a comment on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-690866681
@kaxil that option is not available, because this PR is not under a personal user. Feel free to put up a new PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens edited a comment on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens edited a comment on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-690866681
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens commented on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens commented on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-667958776
The logger responsible is `azure.core.pipeline.policies.http_logging_policy`. Will try add control over it via the config with a default to debug instead of info.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexbegg commented on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
alexbegg commented on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-666748369
Not sure if anyone who uses this noticed but the module `azure.core.pipeline.policies._universal` (which handles logging of HTTP requests for `client.get_secret`) is spitting out a lot of INFO-level logs for every request, no matter if it is a 200 or 404 response.
There might be a better way to stop it from spamming the info logs, but one workaround I found is to change the logging level for the `azure` logger to WARNING, I just not sure how it affects things other than this secret backend. I suggest the following gets added to `azure_key_vault.py`:
```
import logging
logger = logging.getLogger('azure')
logger.setLevel(logging.WARNING)
```
Here is an example of just _**one**_ request shown in the INFO-level logs:
```
{_universal.py:412} INFO - Request URL: '\x1b[1mhttps://**********/secrets/**********/?api-version=REDACTED\x1b[0m'
{_universal.py:413} INFO - Request method: '\x1b[1mGET\x1b[0m'
{_universal.py:414} INFO - Request headers:
{_universal.py:417} INFO - '\x1b[1mAccept\x1b[0m': '\x1b[1mapplication/json\x1b[0m'
{_universal.py:417} INFO - '\x1b[1mx-ms-client-request-id\x1b[0m': '\x1b[1me76166a4-d2b0-11ea-8042-f2d2fd64bfce\x1b[0m'
{_universal.py:417} INFO - '\x1b[1mUser-Agent\x1b[0m': '\x1b[1mazsdk-python-keyvault-secrets/4.1.0 Python/3.6.10 (Linux-4.15.0-1089-azure-x86_64-with-debian-10.3)\x1b[0m'
{_universal.py:417} INFO - '\x1b[1mAuthorization\x1b[0m': '\x1b[1mREDACTED\x1b[0m'
{_universal.py:431} INFO - Response status: 200
{_universal.py:432} INFO - Response headers:
{_universal.py:435} INFO - '\x1b[1mCache-Control\x1b[0m': '\x1b[1mno-cache\x1b[0m'
{_universal.py:435} INFO - '\x1b[1mPragma\x1b[0m': '\x1b[1mno-cache\x1b[0m'
{_universal.py:435} INFO - '\x1b[1mContent-Type\x1b[0m': '\x1b[1mapplication/json; charset=utf-8\x1b[0m'
{_universal.py:435} INFO - '\x1b[1mExpires\x1b[0m': '\x1b[1m-1\x1b[0m'
{_universal.py:435} INFO - '\x1b[1mx-ms-keyvault-region\x1b[0m': '\x1b[1mREDACTED\x1b[0m'
{_universal.py:435} INFO - '\x1b[1mx-ms-request-id\x1b[0m': '\x1b[1maa7b60bd-292a-4f33-a034-21690c75463f\x1b[0m'
{_universal.py:435} INFO - '\x1b[1mx-ms-keyvault-service-version\x1b[0m': '\x1b[1mREDACTED\x1b[0m'
{_universal.py:435} INFO - '\x1b[1mx-ms-keyvault-network-info\x1b[0m': '\x1b[1mconn_type=Ipv4;addr=**********;act_addr_fam=InterNetwork;\x1b[0m'
{_universal.py:435} INFO - '\x1b[1mX-AspNet-Version\x1b[0m': '\x1b[1mREDACTED\x1b[0m'
{_universal.py:435} INFO - '\x1b[1mX-Powered-By\x1b[0m': '\x1b[1mREDACTED\x1b[0m'
{_universal.py:435} INFO - '\x1b[1mStrict-Transport-Security\x1b[0m': '\x1b[1mREDACTED\x1b[0m'
{_universal.py:435} INFO - '\x1b[1mX-Content-Type-Options\x1b[0m': '\x1b[1mREDACTED\x1b[0m'
{_universal.py:435} INFO - '\x1b[1mDate\x1b[0m': '\x1b[1mThu, 30 Jul 2020 22:06:21 GMT\x1b[0m'
{_universal.py:435} INFO - '\x1b[1mContent-Length\x1b[0m': '\x1b[1m355\x1b[0m'
```
P.S. I don't really have the time either to finish this up, but I made a very similar secrets backend for Azure Key Vault that used ClientSecretCredential instead of DefaultAzureCredential, but I like this method better because instead of having to put the client_id, client_secret, and tenant_id in the kwargs you can just add them as environment variables
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexbegg commented on a change in pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
alexbegg commented on a change in pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#discussion_r452378087
##########
File path: airflow/providers/microsoft/azure/secrets/azure_key_vault.py
##########
@@ -0,0 +1,107 @@
+"""
+This module contains a secrets backend for Azure Key Vault.
+"""
+from typing import Optional
+
+from azure.identity import DefaultAzureCredential
+from azure.keyvault.secrets import SecretClient
+from azure.core.exceptions import ResourceNotFoundError
+from cached_property import cached_property
+
+from airflow.secrets import BaseSecretsBackend
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class AzureKeyVaultBackend(BaseSecretsBackend, LoggingMixin):
+ """
+ Retrieves Airflow Connections or Variables from Azure Key Vault secrets.
+
+ The Azure Key Vault can be configred as a secrets backend in the ``airflow.cfg``:
+
+ .. code-block:: ini
+
+ [secrets]
+ backend = airflow.providers.microsoft.azure.secrets.azure_key_vault.AzureKeyVaultBackend
+ backend_kwargs = {"vault_url": "<azure_key_vault_uri>"}
+
+ For example, if the secrets prefix is ``airflow-connections-smtp-default``, this would be accessible
+ if you provide ``{"connections_prefix": "airflow-connections"}`` and request conn_id ``smtp-default``.
+ And if variables prefix is ``airflow-variables-hello``, this would be accessible
+ if you provide ``{"variables_prefix": "airflow-variables"}`` and request variable key ``hello``.
+
+ :param vault_url: The URL of an Azure Key Vault to use
+ :type vault_url: str
+ :param connections_prefix: Specifies the prefix of the secret to read to get Connections
+ :type connections_prefix: str
+ :param variables_prefix: Specifies the prefix of the secret to read to get Variables
+ :type variables_prefix: str
+ :param sep: separator used to concatenate secret_prefix and secret_id. Default: "-"
+ :type sep: str
+ """
+
+ def __init__(self, vault_url: str = None, connections_prefix: str = 'airflow-connections',
+ variables_prefix: str = 'airflow-variables', sep: str = '-', **kwargs):
+ super().__init__(**kwargs)
+ self.connections_prefix = connections_prefix.rstrip(sep)
+ self.variables_prefix = variables_prefix.rstrip(sep)
+ self.vault_url = vault_url
+ self.sep = sep
+ self.kwargs = kwargs
+
+ @cached_property
+ def client(self):
+ """
+ Create a Azure Key Vault client.
+ """
+ credential = DefaultAzureCredential()
Review comment:
I was working on a PR for this too but I was not ready yet, however in my case I used `ClientSecretCredential` from `azure.identity` instead and set up the `backend_kwargs` to include `client_id`, `client_secret`, and `tenant_id`. Would that be better?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens commented on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens commented on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-663843377
@kaxil could use help, yes. Don’t think i can work on this the coming weeks, unfortunately. Feel free to fork this and put up a new PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens closed pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens closed pull request #9639:
URL: https://github.com/apache/airflow/pull/9639
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-656524133
> Should i add config to this backend too?
Yes please, it would allow you to retrieve some of the Airflow configs from Secrets Backend, details here https://airflow.readthedocs.io/en/latest/howto/set-config.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-690739141
@elwinarens We are very close to merging this can you please rebase, or just checkmark "Allow Edit From Maintainers" https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#discussion_r449522855
##########
File path: tests/providers/microsoft/azure/secrets/test_azure_key_vault_backend.py
##########
@@ -0,0 +1,92 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from unittest import TestCase, mock
+
+from moto import mock_secretsmanager
+
+from airflow.providers.microsoft.azure.secrets.azure_key_vault import AzureKeyVaultBackend
+
+
+class TestAzureKeyVaultBackend(TestCase):
+ @mock.patch("airflow.providers.microsoft.azure.secrets.azure_key_vault"
+ "AzureKeyVaultBackend.get_conn_uri")
+ def test_get_connections(self, mock_get_uri):
+ mock_get_uri.return_value = "scheme://user:pass@host:100"
+ conn_list = AzureKeyVaultBackend().get_connections("fake_conn")
+ conn = conn_list[0]
+ assert conn.host == 'host'
+
+ @mock_secretsmanager
+ def test_get_conn_uri(self):
+ param = {
+ 'SecretId': 'airflow-connections-test_postgres',
+ 'SecretString': 'postgresql://airflow:airflow@host:5432/airflow'
+ }
+
+ backend = AzureKeyVaultBackend()
+ backend.client.put_secret_value(**param)
+
+ returned_uri = backend.get_conn_uri(conn_id="test_postgres")
+ self.assertEqual('postgresql://airflow:airflow@host:5432/airflow', returned_uri)
+
+ @mock_secretsmanager
Review comment:
and here and everywhere in the file?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens closed pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens closed pull request #9639:
URL: https://github.com/apache/airflow/pull/9639
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens edited a comment on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens edited a comment on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-667971042
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens edited a comment on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens edited a comment on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-690866681
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens commented on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens commented on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-667872146
@kaxil @alexbegg Will try and make some time this week
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens closed pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens closed pull request #9639:
URL: https://github.com/apache/airflow/pull/9639
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens edited a comment on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens edited a comment on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-663843377
@kaxil Could use help, yes. Don’t think i can work on this the coming weeks, unfortunately. Feel free to fork this and put up a new PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-690557596
ping @elwinarens @flolas
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexbegg commented on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
alexbegg commented on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-690718440
@kaxil If neither are able to continue the work, what is remaining to do here? Does it need a hook? has config secrets been added to this backend yet? I am already using this same backend for my own Airflow installation but as a custom secrets backend as a standalone class, not incorporated into the apache-airflow-backport-providers-microsoft-azure package.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexbegg edited a comment on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
alexbegg edited a comment on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-690718440
@kaxil If neither are able to continue the work, what is remaining to do here? Does it need a hook? has config secrets been added to this backend yet? I am already using this same backend for my own Airflow installation but as a custom secrets backend as a standalone class, not incorporated into the apache-airflow-backport-providers-microsoft-azure package. I was working on developing my own PR for this but I was still working on the hook and other parts when I left off.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens commented on a change in pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens commented on a change in pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#discussion_r452633694
##########
File path: docs/howto/use-alternative-secrets-backend.rst
##########
@@ -429,6 +429,39 @@ When ``gcp_key_path`` is not provided, it will use the Application Default Crede
The value of the Secrets Manager secret id must be the :ref:`connection URI representation <generating_connection_uri>`
of the connection object.
+Azure Key Vault Backend
+^^^^^^^^^^^^^^^^^^^^^^^
+
+To enable the Azure Key Vault as secrets backend, specify :py:class:`~airflow.providers.microsoft.azure.secrets.azure_key_vault.AzureKeyVaultBackend`
+as the ``backend`` in ``[secrets]`` section of ``airflow.cfg``.
+
+Here is a sample configuration:
+
+.. code-block:: ini
+
+ [secrets]
+ backend = airflow.providers.microsoft.azure.secrets.azure_key_vault.AzureKeyVaultBackend
+ backend_kwargs = {"vault_url": "https://example-akv-resource-name.vault.azure.net/", "connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables"}
Review comment:
Nice eye for detail there!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-653479886
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://apache-airflow-slack.herokuapp.com/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens edited a comment on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens edited a comment on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-690866681
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-660960735
Do you need any help in completing this @elwinarens?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens removed a comment on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens removed a comment on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-667958776
The logger responsible is `azure.core.pipeline.policies.http_logging_policy`. Will try add control over it via the config with a default to debug instead of info.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens commented on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens commented on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-690866681
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#discussion_r450807455
##########
File path: airflow/providers/microsoft/azure/secrets/azure_key_vault.py
##########
@@ -0,0 +1,107 @@
+"""
Review comment:
You need to add Apache License to all the new files
```
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#discussion_r450807455
##########
File path: airflow/providers/microsoft/azure/secrets/azure_key_vault.py
##########
@@ -0,0 +1,107 @@
+"""
Review comment:
You need to add Apache License to all the new files at the top
```
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens edited a comment on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens edited a comment on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-667971042
@alexbegg After investigating the verbose logging: while i understand that the logs are verbose and annoying, i don't think the responsibility of changing the default behaviour of the azure python sdk implementation should be changed in this backend.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#discussion_r449521224
##########
File path: airflow/providers/microsoft/azure/secrets/azure_key_vault.py
##########
@@ -0,0 +1,107 @@
+"""
+This module contains a secrets backend for Azure Key Vault.
+"""
+from typing import Optional
+
+from azure.identity import DefaultAzureCredential
+from azure.keyvault.secrets import SecretClient
+from azure.core.exceptions import ResourceNotFoundError
+from cached_property import cached_property
+
+from airflow.secrets import BaseSecretsBackend
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class AzureKeyVaultBackend(BaseSecretsBackend, LoggingMixin):
+ """
+ Retrieves Airflow Connections or Variables from Azure Key Vault secrets.
+
+ The Azure Key Vault can be configred as a secrets backend in the ``airflow.cfg``:
+
+ .. code-block:: ini
+
+ [secrets]
+ backend = airflow.providers.microsoft.azure.secrets.azure_key_vault.AzureKeyVaultBackend
+ backend_kwargs = {"vault_url": "<azure_key_vault_uri>"}
+
+ For example, if the secrets prefix is ``airflow-connections-smtp-default``, this would be accessible
+ if you provide ``{"connections_prefix": "airflow-connections"}`` and request conn_id ``smtp-default``.
+ And if variables prefix is ``airflow-variables-hello``, this would be accessible
+ if you provide ``{"variables_prefix": "airflow-variables"}`` and request variable key ``hello``.
+
+ :param vault_url: The URL of an Azure Key Vault to use
+ :type vault_url: str
+ :param connections_prefix: Specifies the prefix of the secret to read to get Connections
+ :type connections_prefix: str
+ :param variables_prefix: Specifies the prefix of the secret to read to get Variables
+ :type variables_prefix: str
+ :param sep: separator used to concatenate secret_prefix and secret_id. Default: "-"
+ :type sep: str
+ """
+
+ def __init__(self, vault_url: str = None, connections_prefix: str = 'airflow-connections',
+ variables_prefix: str = 'airflow-variables', sep: str = '-', **kwargs):
+ super().__init__(**kwargs)
+ self.connections_prefix = connections_prefix.rstrip(sep)
+ self.variables_prefix = variables_prefix.rstrip(sep)
+ self.vault_url = vault_url
+ self.sep = sep
+ self.kwargs = kwargs
+
+ @cached_property
+ def client(self):
+ """
+ Create a Azure Key Vault client.
+ """
+ credential = DefaultAzureCredential()
Review comment:
What is the limitation of using `DefaultAzureCredential`
https://docs.microsoft.com/en-us/azure/developer/python/azure-sdk-authenticate?tabs=cmd
##########
File path: airflow/providers/microsoft/azure/secrets/azure_key_vault.py
##########
@@ -0,0 +1,107 @@
+"""
+This module contains a secrets backend for Azure Key Vault.
+"""
+from typing import Optional
+
+from azure.identity import DefaultAzureCredential
+from azure.keyvault.secrets import SecretClient
+from azure.core.exceptions import ResourceNotFoundError
+from cached_property import cached_property
+
+from airflow.secrets import BaseSecretsBackend
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class AzureKeyVaultBackend(BaseSecretsBackend, LoggingMixin):
+ """
+ Retrieves Airflow Connections or Variables from Azure Key Vault secrets.
+
+ The Azure Key Vault can be configred as a secrets backend in the ``airflow.cfg``:
+
+ .. code-block:: ini
+
+ [secrets]
+ backend = airflow.providers.microsoft.azure.secrets.azure_key_vault.AzureKeyVaultBackend
+ backend_kwargs = {"vault_url": "<azure_key_vault_uri>"}
+
+ For example, if the secrets prefix is ``airflow-connections-smtp-default``, this would be accessible
+ if you provide ``{"connections_prefix": "airflow-connections"}`` and request conn_id ``smtp-default``.
+ And if variables prefix is ``airflow-variables-hello``, this would be accessible
+ if you provide ``{"variables_prefix": "airflow-variables"}`` and request variable key ``hello``.
+
+ :param vault_url: The URL of an Azure Key Vault to use
+ :type vault_url: str
+ :param connections_prefix: Specifies the prefix of the secret to read to get Connections
+ :type connections_prefix: str
+ :param variables_prefix: Specifies the prefix of the secret to read to get Variables
+ :type variables_prefix: str
+ :param sep: separator used to concatenate secret_prefix and secret_id. Default: "-"
+ :type sep: str
+ """
+
+ def __init__(self, vault_url: str = None, connections_prefix: str = 'airflow-connections',
+ variables_prefix: str = 'airflow-variables', sep: str = '-', **kwargs):
+ super().__init__(**kwargs)
Review comment:
```suggestion
super().__init__()
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#discussion_r450807727
##########
File path: airflow/providers/microsoft/azure/secrets/azure_key_vault.py
##########
@@ -0,0 +1,107 @@
+"""
Review comment:
```suggestion
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexbegg commented on a change in pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
alexbegg commented on a change in pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#discussion_r452380634
##########
File path: airflow/providers/microsoft/azure/secrets/azure_key_vault.py
##########
@@ -0,0 +1,120 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Optional
+
+from azure.core.exceptions import ResourceNotFoundError
+from azure.identity import DefaultAzureCredential
+from azure.keyvault.secrets import SecretClient
+from cached_property import cached_property
+
+from airflow.secrets import BaseSecretsBackend
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class AzureKeyVaultBackend(BaseSecretsBackend, LoggingMixin):
+ """
+ Retrieves Airflow Connections or Variables from Azure Key Vault secrets.
+
+ The Azure Key Vault can be configred as a secrets backend in the ``airflow.cfg``:
+
+ .. code-block:: ini
+
+ [secrets]
+ backend = airflow.providers.microsoft.azure.secrets.azure_key_vault.AzureKeyVaultBackend
+ backend_kwargs = {"vault_url": "<azure_key_vault_uri>"}
+
+ For example, if the secrets prefix is ``airflow-connections-smtp-default``, this would be accessible
+ if you provide ``{"connections_prefix": "airflow-connections"}`` and request conn_id ``smtp-default``.
+ And if variables prefix is ``airflow-variables-hello``, this would be accessible
+ if you provide ``{"variables_prefix": "airflow-variables"}`` and request variable key ``hello``.
+
+ :param vault_url: The URL of an Azure Key Vault to use
+ :type vault_url: str
+ :param connections_prefix: Specifies the prefix of the secret to read to get Connections
+ :type connections_prefix: str
+ :param variables_prefix: Specifies the prefix of the secret to read to get Variables
+ :type variables_prefix: str
+ :param sep: separator used to concatenate secret_prefix and secret_id. Default: "-"
+ :type sep: str
+ """
+
+ def __init__(self, vault_url: str = '', connections_prefix: str = 'airflow-connections',
+ variables_prefix: str = 'airflow-variables', sep: str = '-', **kwargs):
+ super().__init__()
+ self.vault_url = vault_url
Review comment:
I think we should follow the pattern of the other secrets backend have `connections_prefix` and `variables_prefix` as the first 2 kwargs and move `vault_url` after those two.
##########
File path: airflow/providers/microsoft/azure/secrets/azure_key_vault.py
##########
@@ -0,0 +1,120 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Optional
+
+from azure.core.exceptions import ResourceNotFoundError
+from azure.identity import DefaultAzureCredential
+from azure.keyvault.secrets import SecretClient
+from cached_property import cached_property
+
+from airflow.secrets import BaseSecretsBackend
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class AzureKeyVaultBackend(BaseSecretsBackend, LoggingMixin):
+ """
+ Retrieves Airflow Connections or Variables from Azure Key Vault secrets.
+
+ The Azure Key Vault can be configred as a secrets backend in the ``airflow.cfg``:
+
+ .. code-block:: ini
+
+ [secrets]
+ backend = airflow.providers.microsoft.azure.secrets.azure_key_vault.AzureKeyVaultBackend
+ backend_kwargs = {"vault_url": "<azure_key_vault_uri>"}
Review comment:
The example of backend_kwargs should be
`{"connections_prefix": "airflow-connections", "vault_url": "<azure_key_vault_uri>"}`
to include an example of using a prefix, like the other secrets backends use
##########
File path: docs/howto/use-alternative-secrets-backend.rst
##########
@@ -429,6 +429,39 @@ When ``gcp_key_path`` is not provided, it will use the Application Default Crede
The value of the Secrets Manager secret id must be the :ref:`connection URI representation <generating_connection_uri>`
of the connection object.
+Azure Key Vault Backend
+^^^^^^^^^^^^^^^^^^^^^^^
+
+To enable the Azure Key Vault as secrets backend, specify :py:class:`~airflow.providers.microsoft.azure.secrets.azure_key_vault.AzureKeyVaultBackend`
+as the ``backend`` in ``[secrets]`` section of ``airflow.cfg``.
+
+Here is a sample configuration:
+
+.. code-block:: ini
+
+ [secrets]
+ backend = airflow.providers.microsoft.azure.secrets.azure_key_vault.AzureKeyVaultBackend
+ backend_kwargs = {"vault_url": "https://example-akv-resource-name.vault.azure.net/", "connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables"}
Review comment:
Same as I commented in the python file, `connections_prefix` and `variables_prefix` should go before `vault_url` to be consistent with the rest of the secrets backend
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#discussion_r452664124
##########
File path: airflow/providers/microsoft/azure/secrets/azure_key_vault.py
##########
@@ -0,0 +1,131 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Optional
+
+from azure.core.exceptions import ResourceNotFoundError
+from azure.identity import DefaultAzureCredential
+from azure.keyvault.secrets import SecretClient
+from cached_property import cached_property
+
+from airflow.secrets import BaseSecretsBackend
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class AzureKeyVaultBackend(BaseSecretsBackend, LoggingMixin):
+ """
+ Retrieves Airflow Connections or Variables from Azure Key Vault secrets.
+
+ The Azure Key Vault can be configred as a secrets backend in the ``airflow.cfg``:
+
+ .. code-block:: ini
+
+ [secrets]
+ backend = airflow.providers.microsoft.azure.secrets.azure_key_vault.AzureKeyVaultBackend
+ backend_kwargs = {"connections_prefix": "airflow-connections", "vault_url": "<azure_key_vault_uri>"}
+
+ For example, if the secrets prefix is ``airflow-connections-smtp-default``, this would be accessible
+ if you provide ``{"connections_prefix": "airflow-connections"}`` and request conn_id ``smtp-default``.
+ And if variables prefix is ``airflow-variables-hello``, this would be accessible
+ if you provide ``{"variables_prefix": "airflow-variables"}`` and request variable key ``hello``.
+
+ :param vault_url: The URL of an Azure Key Vault to use
+ :type vault_url: str
+ :param connections_prefix: Specifies the prefix of the secret to read to get Connections
+ :type connections_prefix: str
+ :param variables_prefix: Specifies the prefix of the secret to read to get Variables
+ :type variables_prefix: str
+ :param sep: separator used to concatenate secret_prefix and secret_id. Default: "-"
+ :type sep: str
+ """
+
+ def __init__(
+ self,
+ connections_prefix: str = 'airflow-connections',
+ variables_prefix: str = 'airflow-variables',
+ vault_url: str = '',
+ sep: str = '-',
+ **kwargs
+ ):
+ super().__init__()
+ self.vault_url = vault_url
+ self.connections_prefix = connections_prefix.rstrip(sep)
+ self.variables_prefix = variables_prefix.rstrip(sep)
+ self.sep = sep
+ self.kwargs = kwargs
+
+ @cached_property
+ def client(self):
+ """
+ Create a Azure Key Vault client.
+ """
+ credential = DefaultAzureCredential()
+ client = SecretClient(
+ vault_url=self.vault_url,
+ credential=credential,
+ **self.kwargs
+ )
+ return client
+
+ def get_conn_uri(self, conn_id: str) -> Optional[str]:
+ """
+ Get an Airflow Connection URI from an Azure Key Vault secret
+
+ :param conn_id: The Airflow connection id to retrieve
+ :type conn_id: str
+ """
+ name = self.build_path(self.connections_prefix, conn_id, self.sep)
+ return self.get_secret_value(name)
+
+ def get_variable(self, key: str) -> Optional[str]:
+ """
+ Get an Airflow Variable from an Azure Key Vault secret.
+
+ :param key: Variable Key
+ :type key: str
+ :return: Variable Value
+ """
+ name = self.build_path(self.variables_prefix, key, self.sep)
+ return self.get_secret_value(name)
+
+ @staticmethod
+ def build_path(path_prefix: str, secret_id: str, sep: str = '-') -> str:
+ """
+ Given a path_prefix and secret_id, build a valid secret name for the Azure Key Vault Backend.
+ Also replaces underscore in the path with dashes to support easy switching between
+ environmentvariables, so ``connection_default`` becomes ``connection-default``.
+
+ :param path_prefix: The path prefix of the secret to retrieve
+ :type path_prefix: str
+ :param secret_id: Name of the secret
+ :type secret_id: str
+ :param sep: Separator used to concatenate path_prefix and secret_id
+ :type sep: str
+ """
+ path = f'{path_prefix}{sep}{secret_id}'
+ return path.replace('_', sep)
+
+ def get_secret_value(self, name: str) -> Optional[str]:
Review comment:
We will have hooks for that, for example: https://github.com/apache/airflow/blob/master/airflow/providers/hashicorp/hooks/vault.py
WDYT ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexbegg commented on a change in pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
alexbegg commented on a change in pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#discussion_r452709383
##########
File path: airflow/providers/microsoft/azure/secrets/azure_key_vault.py
##########
@@ -0,0 +1,131 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Optional
+
+from azure.core.exceptions import ResourceNotFoundError
+from azure.identity import DefaultAzureCredential
+from azure.keyvault.secrets import SecretClient
+from cached_property import cached_property
+
+from airflow.secrets import BaseSecretsBackend
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class AzureKeyVaultBackend(BaseSecretsBackend, LoggingMixin):
+ """
+ Retrieves Airflow Connections or Variables from Azure Key Vault secrets.
+
+ The Azure Key Vault can be configred as a secrets backend in the ``airflow.cfg``:
+
+ .. code-block:: ini
+
+ [secrets]
+ backend = airflow.providers.microsoft.azure.secrets.azure_key_vault.AzureKeyVaultBackend
+ backend_kwargs = {"connections_prefix": "airflow-connections", "vault_url": "<azure_key_vault_uri>"}
+
+ For example, if the secrets prefix is ``airflow-connections-smtp-default``, this would be accessible
+ if you provide ``{"connections_prefix": "airflow-connections"}`` and request conn_id ``smtp-default``.
+ And if variables prefix is ``airflow-variables-hello``, this would be accessible
+ if you provide ``{"variables_prefix": "airflow-variables"}`` and request variable key ``hello``.
+
+ :param vault_url: The URL of an Azure Key Vault to use
Review comment:
One more: also put vault_url after variables_prefix here too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens commented on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens commented on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-690866681
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] flolas commented on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
flolas commented on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-676521708
I'm interested in this functionality. Do you need help finishing the PR?
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens edited a comment on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens edited a comment on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-690866681
@kaxil that option is not available, because this PR is not under a personal user. Feel free cherry pick and to put up a new PR. Let me know if you need anything.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens commented on a change in pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens commented on a change in pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#discussion_r454221466
##########
File path: airflow/providers/microsoft/azure/secrets/azure_key_vault.py
##########
@@ -0,0 +1,131 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Optional
+
+from azure.core.exceptions import ResourceNotFoundError
+from azure.identity import DefaultAzureCredential
+from azure.keyvault.secrets import SecretClient
+from cached_property import cached_property
+
+from airflow.secrets import BaseSecretsBackend
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class AzureKeyVaultBackend(BaseSecretsBackend, LoggingMixin):
+ """
+ Retrieves Airflow Connections or Variables from Azure Key Vault secrets.
+
+ The Azure Key Vault can be configred as a secrets backend in the ``airflow.cfg``:
+
+ .. code-block:: ini
+
+ [secrets]
+ backend = airflow.providers.microsoft.azure.secrets.azure_key_vault.AzureKeyVaultBackend
+ backend_kwargs = {"connections_prefix": "airflow-connections", "vault_url": "<azure_key_vault_uri>"}
+
+ For example, if the secrets prefix is ``airflow-connections-smtp-default``, this would be accessible
+ if you provide ``{"connections_prefix": "airflow-connections"}`` and request conn_id ``smtp-default``.
+ And if variables prefix is ``airflow-variables-hello``, this would be accessible
+ if you provide ``{"variables_prefix": "airflow-variables"}`` and request variable key ``hello``.
+
+ :param vault_url: The URL of an Azure Key Vault to use
+ :type vault_url: str
+ :param connections_prefix: Specifies the prefix of the secret to read to get Connections
+ :type connections_prefix: str
+ :param variables_prefix: Specifies the prefix of the secret to read to get Variables
+ :type variables_prefix: str
+ :param sep: separator used to concatenate secret_prefix and secret_id. Default: "-"
+ :type sep: str
+ """
+
+ def __init__(
+ self,
+ connections_prefix: str = 'airflow-connections',
+ variables_prefix: str = 'airflow-variables',
+ vault_url: str = '',
+ sep: str = '-',
+ **kwargs
+ ):
+ super().__init__()
+ self.vault_url = vault_url
+ self.connections_prefix = connections_prefix.rstrip(sep)
+ self.variables_prefix = variables_prefix.rstrip(sep)
+ self.sep = sep
+ self.kwargs = kwargs
+
+ @cached_property
+ def client(self):
+ """
+ Create a Azure Key Vault client.
+ """
+ credential = DefaultAzureCredential()
+ client = SecretClient(
+ vault_url=self.vault_url,
+ credential=credential,
+ **self.kwargs
+ )
+ return client
+
+ def get_conn_uri(self, conn_id: str) -> Optional[str]:
+ """
+ Get an Airflow Connection URI from an Azure Key Vault secret
+
+ :param conn_id: The Airflow connection id to retrieve
+ :type conn_id: str
+ """
+ name = self.build_path(self.connections_prefix, conn_id, self.sep)
+ return self.get_secret_value(name)
+
+ def get_variable(self, key: str) -> Optional[str]:
+ """
+ Get an Airflow Variable from an Azure Key Vault secret.
+
+ :param key: Variable Key
+ :type key: str
+ :return: Variable Value
+ """
+ name = self.build_path(self.variables_prefix, key, self.sep)
+ return self.get_secret_value(name)
+
+ @staticmethod
+ def build_path(path_prefix: str, secret_id: str, sep: str = '-') -> str:
+ """
+ Given a path_prefix and secret_id, build a valid secret name for the Azure Key Vault Backend.
+ Also replaces underscore in the path with dashes to support easy switching between
+ environmentvariables, so ``connection_default`` becomes ``connection-default``.
+
+ :param path_prefix: The path prefix of the secret to retrieve
+ :type path_prefix: str
+ :param secret_id: Name of the secret
+ :type secret_id: str
+ :param sep: Separator used to concatenate path_prefix and secret_id
+ :type sep: str
+ """
+ path = f'{path_prefix}{sep}{secret_id}'
+ return path.replace('_', sep)
+
+ def get_secret_value(self, name: str) -> Optional[str]:
Review comment:
I will make it a private method again and will create a hook later one.
##########
File path: airflow/providers/microsoft/azure/secrets/azure_key_vault.py
##########
@@ -0,0 +1,131 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Optional
+
+from azure.core.exceptions import ResourceNotFoundError
+from azure.identity import DefaultAzureCredential
+from azure.keyvault.secrets import SecretClient
+from cached_property import cached_property
+
+from airflow.secrets import BaseSecretsBackend
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class AzureKeyVaultBackend(BaseSecretsBackend, LoggingMixin):
+ """
+ Retrieves Airflow Connections or Variables from Azure Key Vault secrets.
+
+ The Azure Key Vault can be configred as a secrets backend in the ``airflow.cfg``:
+
+ .. code-block:: ini
+
+ [secrets]
+ backend = airflow.providers.microsoft.azure.secrets.azure_key_vault.AzureKeyVaultBackend
+ backend_kwargs = {"connections_prefix": "airflow-connections", "vault_url": "<azure_key_vault_uri>"}
+
+ For example, if the secrets prefix is ``airflow-connections-smtp-default``, this would be accessible
+ if you provide ``{"connections_prefix": "airflow-connections"}`` and request conn_id ``smtp-default``.
+ And if variables prefix is ``airflow-variables-hello``, this would be accessible
+ if you provide ``{"variables_prefix": "airflow-variables"}`` and request variable key ``hello``.
+
+ :param vault_url: The URL of an Azure Key Vault to use
+ :type vault_url: str
+ :param connections_prefix: Specifies the prefix of the secret to read to get Connections
+ :type connections_prefix: str
+ :param variables_prefix: Specifies the prefix of the secret to read to get Variables
+ :type variables_prefix: str
+ :param sep: separator used to concatenate secret_prefix and secret_id. Default: "-"
+ :type sep: str
+ """
+
+ def __init__(
+ self,
+ connections_prefix: str = 'airflow-connections',
+ variables_prefix: str = 'airflow-variables',
+ vault_url: str = '',
+ sep: str = '-',
+ **kwargs
+ ):
+ super().__init__()
+ self.vault_url = vault_url
+ self.connections_prefix = connections_prefix.rstrip(sep)
+ self.variables_prefix = variables_prefix.rstrip(sep)
+ self.sep = sep
+ self.kwargs = kwargs
+
+ @cached_property
+ def client(self):
+ """
+ Create a Azure Key Vault client.
+ """
+ credential = DefaultAzureCredential()
+ client = SecretClient(
+ vault_url=self.vault_url,
+ credential=credential,
+ **self.kwargs
+ )
+ return client
+
+ def get_conn_uri(self, conn_id: str) -> Optional[str]:
+ """
+ Get an Airflow Connection URI from an Azure Key Vault secret
+
+ :param conn_id: The Airflow connection id to retrieve
+ :type conn_id: str
+ """
+ name = self.build_path(self.connections_prefix, conn_id, self.sep)
+ return self.get_secret_value(name)
+
+ def get_variable(self, key: str) -> Optional[str]:
+ """
+ Get an Airflow Variable from an Azure Key Vault secret.
+
+ :param key: Variable Key
+ :type key: str
+ :return: Variable Value
+ """
+ name = self.build_path(self.variables_prefix, key, self.sep)
+ return self.get_secret_value(name)
+
+ @staticmethod
+ def build_path(path_prefix: str, secret_id: str, sep: str = '-') -> str:
+ """
+ Given a path_prefix and secret_id, build a valid secret name for the Azure Key Vault Backend.
+ Also replaces underscore in the path with dashes to support easy switching between
+ environmentvariables, so ``connection_default`` becomes ``connection-default``.
+
+ :param path_prefix: The path prefix of the secret to retrieve
+ :type path_prefix: str
+ :param secret_id: Name of the secret
+ :type secret_id: str
+ :param sep: Separator used to concatenate path_prefix and secret_id
+ :type sep: str
+ """
+ path = f'{path_prefix}{sep}{secret_id}'
+ return path.replace('_', sep)
+
+ def get_secret_value(self, name: str) -> Optional[str]:
Review comment:
I will make it a private method again and will create a hook later on.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens commented on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens commented on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-667971042
> http_logging_policy
@alexbegg While i understand that the logs are verbose and annoying, i don't think the responsibility of changing the default behaviour of the azure python sdk implementation should be changed in this backend.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexbegg commented on a change in pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
alexbegg commented on a change in pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#discussion_r452388036
##########
File path: airflow/providers/microsoft/azure/secrets/__init__.py
##########
@@ -0,0 +1,16 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
Review comment:
Add a new line at the end of the file
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#discussion_r449519179
##########
File path: airflow/providers/microsoft/azure/secrets/azure_key_vault.py
##########
@@ -0,0 +1,107 @@
+"""
+This module contains a secrets backend for Azure Key Vault.
+"""
+from typing import Optional
+
+from azure.identity import DefaultAzureCredential
+from azure.keyvault.secrets import SecretClient
+from azure.core.exceptions import ResourceNotFoundError
+from cached_property import cached_property
+
+from airflow.secrets import BaseSecretsBackend
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class AzureKeyVaultBackend(BaseSecretsBackend, LoggingMixin):
+ """
+ Retrieves Airflow Connections or Variables from Azure Key Vault secrets.
+
+ The Azure Key Vault can be configred as a secrets backend in the ``airflow.cfg``:
+
+ .. code-block:: ini
+
+ [secrets]
+ backend = airflow.providers.microsoft.azure.secrets.azure_key_vault.AzureKeyVaultBackend
+ backend_kwargs = {"vault_url": "<azure_key_vault_uri>"}
+
+ For example, if the secrets prefix is ``airflow-connections-smtp-default``, this would be accessible
+ if you provide ``{"connections_prefix": "airflow-connections"}`` and request conn_id ``smtp-default``.
+ And if variables prefix is ``airflow-variables-hello``, this would be accessible
+ if you provide ``{"variables_prefix": "airflow-variables"}`` and request variable key ``hello``.
+
+ :param vault_url: The URL of an Azure Key Vault to use
+ :type vault_url: str
+ :param connections_prefix: Specifies the prefix of the secret to read to get Connections
+ :type connections_prefix: str
+ :param variables_prefix: Specifies the prefix of the secret to read to get Variables
+ :type variables_prefix: str
+ :param sep: separator used to concatenate secret_prefix and secret_id. Default: "-"
+ :type sep: str
+ """
+
+ def __init__(self, vault_url: str = None, connections_prefix: str = 'airflow-connections',
+ variables_prefix: str = 'airflow-variables', sep: str = '-', **kwargs):
+ super().__init__(**kwargs)
+ self.connections_prefix = connections_prefix.rstrip(sep)
+ self.variables_prefix = variables_prefix.rstrip(sep)
+ self.vault_url = vault_url
+ self.sep = sep
+ self.kwargs = kwargs
+
+ @cached_property
+ def client(self):
+ """
+ Create a Azure Key Vault client.
+ """
+ credential = DefaultAzureCredential()
+ client = SecretClient(
+ vault_url=self.vault_url,
+ credential=credential,
+ **self.kwargs
+ )
+ return client
+
+ def get_conn_uri(self, conn_id: str) -> Optional[str]:
+ """
+ Get an Airflow Connection URI from an Azure Key Vault secret
+
+ :param conn_id: The Airflow connection id to retrieve
+ :type conn_id: str
+ """
+ return self._get_secret(self.connections_prefix, conn_id)
+
+ def get_variable(self, key: str) -> Optional[str]:
+ """
+ Get an Airflow Variable from an Azure Key Vault secret.
+
+ :param key: Variable Key
+ :type key: str
+ :return: Variable Value
+ """
+ return self._get_secret(self.variables_prefix, key)
+
+ @staticmethod
+ def build_path(path_prefix: str, secret_id: str, sep: str = '-') -> str:
+ """
+ Given a path_prefix and secret_id, build a valid secret name for the Azure Key Vault Backend.
+ Also replaces underscore in the path with dashes to support easy switching between
+ environmentvariables, so ``connection_default`` becomes ``connection-default``.
+
+ :param path_prefix: The path prefix of the secret to retrieve
+ :type path_prefix: str
+ :param secret_id: Name of the secret
+ :type secret_id: str
+ :param sep: Separator used to concatenate path_prefix and secret_id
+ :type sep: str
+ """
+ path = '{}{}{}'.format(path_prefix, sep, secret_id)
Review comment:
```suggestion
path = f'{path_prefix}{sep}{secret_id}'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens commented on a change in pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens commented on a change in pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#discussion_r449549570
##########
File path: tests/providers/microsoft/azure/secrets/test_azure_key_vault_backend.py
##########
@@ -0,0 +1,92 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from unittest import TestCase, mock
+
+from moto import mock_secretsmanager
Review comment:
Nice catch, copy pasta from aws. Will fix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-690732567
> @kaxil If neither are able to continue the work, what is remaining to do here? Does it need a hook? has config secrets been added to this backend yet? I am already using this same backend for my own Airflow installation but as a custom secrets backend as a standalone class, not incorporated into the apache-airflow-backport-providers-microsoft-azure package. I was working on developing my own PR for this but I was still working on the hook and other parts when I left off.
It needs rebase on Master, I tried but the PR author hasn't allowed "Edit from Maintainers"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens commented on a change in pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens commented on a change in pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#discussion_r449562999
##########
File path: airflow/providers/microsoft/azure/secrets/azure_key_vault.py
##########
@@ -0,0 +1,107 @@
+"""
+This module contains a secrets backend for Azure Key Vault.
+"""
+from typing import Optional
+
+from azure.identity import DefaultAzureCredential
+from azure.keyvault.secrets import SecretClient
+from azure.core.exceptions import ResourceNotFoundError
+from cached_property import cached_property
+
+from airflow.secrets import BaseSecretsBackend
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class AzureKeyVaultBackend(BaseSecretsBackend, LoggingMixin):
+ """
+ Retrieves Airflow Connections or Variables from Azure Key Vault secrets.
+
+ The Azure Key Vault can be configred as a secrets backend in the ``airflow.cfg``:
+
+ .. code-block:: ini
+
+ [secrets]
+ backend = airflow.providers.microsoft.azure.secrets.azure_key_vault.AzureKeyVaultBackend
+ backend_kwargs = {"vault_url": "<azure_key_vault_uri>"}
+
+ For example, if the secrets prefix is ``airflow-connections-smtp-default``, this would be accessible
+ if you provide ``{"connections_prefix": "airflow-connections"}`` and request conn_id ``smtp-default``.
+ And if variables prefix is ``airflow-variables-hello``, this would be accessible
+ if you provide ``{"variables_prefix": "airflow-variables"}`` and request variable key ``hello``.
+
+ :param vault_url: The URL of an Azure Key Vault to use
+ :type vault_url: str
+ :param connections_prefix: Specifies the prefix of the secret to read to get Connections
+ :type connections_prefix: str
+ :param variables_prefix: Specifies the prefix of the secret to read to get Variables
+ :type variables_prefix: str
+ :param sep: separator used to concatenate secret_prefix and secret_id. Default: "-"
+ :type sep: str
+ """
+
+ def __init__(self, vault_url: str = None, connections_prefix: str = 'airflow-connections',
+ variables_prefix: str = 'airflow-variables', sep: str = '-', **kwargs):
+ super().__init__(**kwargs)
+ self.connections_prefix = connections_prefix.rstrip(sep)
+ self.variables_prefix = variables_prefix.rstrip(sep)
+ self.vault_url = vault_url
+ self.sep = sep
+ self.kwargs = kwargs
+
+ @cached_property
+ def client(self):
+ """
+ Create a Azure Key Vault client.
+ """
+ credential = DefaultAzureCredential()
Review comment:
More details https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/identity/azure-identity#defaultazurecredential
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens commented on a change in pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens commented on a change in pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#discussion_r452634688
##########
File path: airflow/providers/microsoft/azure/secrets/azure_key_vault.py
##########
@@ -0,0 +1,131 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Optional
+
+from azure.core.exceptions import ResourceNotFoundError
+from azure.identity import DefaultAzureCredential
+from azure.keyvault.secrets import SecretClient
+from cached_property import cached_property
+
+from airflow.secrets import BaseSecretsBackend
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class AzureKeyVaultBackend(BaseSecretsBackend, LoggingMixin):
+ """
+ Retrieves Airflow Connections or Variables from Azure Key Vault secrets.
+
+ The Azure Key Vault can be configred as a secrets backend in the ``airflow.cfg``:
+
+ .. code-block:: ini
+
+ [secrets]
+ backend = airflow.providers.microsoft.azure.secrets.azure_key_vault.AzureKeyVaultBackend
+ backend_kwargs = {"connections_prefix": "airflow-connections", "vault_url": "<azure_key_vault_uri>"}
+
+ For example, if the secrets prefix is ``airflow-connections-smtp-default``, this would be accessible
+ if you provide ``{"connections_prefix": "airflow-connections"}`` and request conn_id ``smtp-default``.
+ And if variables prefix is ``airflow-variables-hello``, this would be accessible
+ if you provide ``{"variables_prefix": "airflow-variables"}`` and request variable key ``hello``.
+
+ :param vault_url: The URL of an Azure Key Vault to use
+ :type vault_url: str
+ :param connections_prefix: Specifies the prefix of the secret to read to get Connections
+ :type connections_prefix: str
+ :param variables_prefix: Specifies the prefix of the secret to read to get Variables
+ :type variables_prefix: str
+ :param sep: separator used to concatenate secret_prefix and secret_id. Default: "-"
+ :type sep: str
+ """
+
+ def __init__(
+ self,
+ connections_prefix: str = 'airflow-connections',
+ variables_prefix: str = 'airflow-variables',
+ vault_url: str = '',
+ sep: str = '-',
+ **kwargs
+ ):
+ super().__init__()
+ self.vault_url = vault_url
+ self.connections_prefix = connections_prefix.rstrip(sep)
+ self.variables_prefix = variables_prefix.rstrip(sep)
+ self.sep = sep
+ self.kwargs = kwargs
+
+ @cached_property
+ def client(self):
+ """
+ Create a Azure Key Vault client.
+ """
+ credential = DefaultAzureCredential()
+ client = SecretClient(
+ vault_url=self.vault_url,
+ credential=credential,
+ **self.kwargs
+ )
+ return client
+
+ def get_conn_uri(self, conn_id: str) -> Optional[str]:
+ """
+ Get an Airflow Connection URI from an Azure Key Vault secret
+
+ :param conn_id: The Airflow connection id to retrieve
+ :type conn_id: str
+ """
+ name = self.build_path(self.connections_prefix, conn_id, self.sep)
+ return self.get_secret_value(name)
+
+ def get_variable(self, key: str) -> Optional[str]:
+ """
+ Get an Airflow Variable from an Azure Key Vault secret.
+
+ :param key: Variable Key
+ :type key: str
+ :return: Variable Value
+ """
+ name = self.build_path(self.variables_prefix, key, self.sep)
+ return self.get_secret_value(name)
+
+ @staticmethod
+ def build_path(path_prefix: str, secret_id: str, sep: str = '-') -> str:
+ """
+ Given a path_prefix and secret_id, build a valid secret name for the Azure Key Vault Backend.
+ Also replaces underscore in the path with dashes to support easy switching between
+ environmentvariables, so ``connection_default`` becomes ``connection-default``.
+
+ :param path_prefix: The path prefix of the secret to retrieve
+ :type path_prefix: str
+ :param secret_id: Name of the secret
+ :type secret_id: str
+ :param sep: Separator used to concatenate path_prefix and secret_id
+ :type sep: str
+ """
+ path = f'{path_prefix}{sep}{secret_id}'
+ return path.replace('_', sep)
+
+ def get_secret_value(self, name: str) -> Optional[str]:
Review comment:
I've refactored _get_secret to get_secret_value to allow use of this backend for secrets that do not conform to the prefixes provided for connections and variables. We have a use case for this as secret names are sometimes internally created by another department that conform to different naming standards.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens commented on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens commented on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-656494910
@kaxil In the aws secrets manager in the docstring it says:
```
:param config_prefix: Specifies the prefix of the secret to read to get Variables.
:type config_prefix: str
```
(Variables should be Config i think)
Should i add config to this backend too?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens closed pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens closed pull request #9639:
URL: https://github.com/apache/airflow/pull/9639
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens edited a comment on pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens edited a comment on pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#issuecomment-690866681
@kaxil that option is not available, because this PR is not under a personal user. Feel free to cherry pick and to put up a new PR. Let me know if you need anything.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#discussion_r449522757
##########
File path: tests/providers/microsoft/azure/secrets/test_azure_key_vault_backend.py
##########
@@ -0,0 +1,92 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from unittest import TestCase, mock
+
+from moto import mock_secretsmanager
Review comment:
Why do we use `moto` here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elwinarens commented on a change in pull request #9639: Add secrets backend for microsoft azure key vault
Posted by GitBox <gi...@apache.org>.
elwinarens commented on a change in pull request #9639:
URL: https://github.com/apache/airflow/pull/9639#discussion_r449549782
##########
File path: airflow/providers/microsoft/azure/secrets/azure_key_vault.py
##########
@@ -0,0 +1,107 @@
+"""
+This module contains a secrets backend for Azure Key Vault.
+"""
+from typing import Optional
+
+from azure.identity import DefaultAzureCredential
+from azure.keyvault.secrets import SecretClient
+from azure.core.exceptions import ResourceNotFoundError
+from cached_property import cached_property
+
+from airflow.secrets import BaseSecretsBackend
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class AzureKeyVaultBackend(BaseSecretsBackend, LoggingMixin):
+ """
+ Retrieves Airflow Connections or Variables from Azure Key Vault secrets.
+
+ The Azure Key Vault can be configred as a secrets backend in the ``airflow.cfg``:
+
+ .. code-block:: ini
+
+ [secrets]
+ backend = airflow.providers.microsoft.azure.secrets.azure_key_vault.AzureKeyVaultBackend
+ backend_kwargs = {"vault_url": "<azure_key_vault_uri>"}
+
+ For example, if the secrets prefix is ``airflow-connections-smtp-default``, this would be accessible
+ if you provide ``{"connections_prefix": "airflow-connections"}`` and request conn_id ``smtp-default``.
+ And if variables prefix is ``airflow-variables-hello``, this would be accessible
+ if you provide ``{"variables_prefix": "airflow-variables"}`` and request variable key ``hello``.
+
+ :param vault_url: The URL of an Azure Key Vault to use
+ :type vault_url: str
+ :param connections_prefix: Specifies the prefix of the secret to read to get Connections
+ :type connections_prefix: str
+ :param variables_prefix: Specifies the prefix of the secret to read to get Variables
+ :type variables_prefix: str
+ :param sep: separator used to concatenate secret_prefix and secret_id. Default: "-"
+ :type sep: str
+ """
+
+ def __init__(self, vault_url: str = None, connections_prefix: str = 'airflow-connections',
+ variables_prefix: str = 'airflow-variables', sep: str = '-', **kwargs):
+ super().__init__(**kwargs)
+ self.connections_prefix = connections_prefix.rstrip(sep)
+ self.variables_prefix = variables_prefix.rstrip(sep)
+ self.vault_url = vault_url
+ self.sep = sep
+ self.kwargs = kwargs
+
+ @cached_property
+ def client(self):
+ """
+ Create a Azure Key Vault client.
+ """
+ credential = DefaultAzureCredential()
Review comment:
Would you mind elaborating?
##########
File path: airflow/providers/microsoft/azure/secrets/azure_key_vault.py
##########
@@ -0,0 +1,107 @@
+"""
+This module contains a secrets backend for Azure Key Vault.
+"""
+from typing import Optional
+
+from azure.identity import DefaultAzureCredential
+from azure.keyvault.secrets import SecretClient
+from azure.core.exceptions import ResourceNotFoundError
+from cached_property import cached_property
+
+from airflow.secrets import BaseSecretsBackend
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class AzureKeyVaultBackend(BaseSecretsBackend, LoggingMixin):
+ """
+ Retrieves Airflow Connections or Variables from Azure Key Vault secrets.
+
+ The Azure Key Vault can be configred as a secrets backend in the ``airflow.cfg``:
+
+ .. code-block:: ini
+
+ [secrets]
+ backend = airflow.providers.microsoft.azure.secrets.azure_key_vault.AzureKeyVaultBackend
+ backend_kwargs = {"vault_url": "<azure_key_vault_uri>"}
+
+ For example, if the secrets prefix is ``airflow-connections-smtp-default``, this would be accessible
+ if you provide ``{"connections_prefix": "airflow-connections"}`` and request conn_id ``smtp-default``.
+ And if variables prefix is ``airflow-variables-hello``, this would be accessible
+ if you provide ``{"variables_prefix": "airflow-variables"}`` and request variable key ``hello``.
+
+ :param vault_url: The URL of an Azure Key Vault to use
+ :type vault_url: str
+ :param connections_prefix: Specifies the prefix of the secret to read to get Connections
+ :type connections_prefix: str
+ :param variables_prefix: Specifies the prefix of the secret to read to get Variables
+ :type variables_prefix: str
+ :param sep: separator used to concatenate secret_prefix and secret_id. Default: "-"
+ :type sep: str
+ """
+
+ def __init__(self, vault_url: str = None, connections_prefix: str = 'airflow-connections',
+ variables_prefix: str = 'airflow-variables', sep: str = '-', **kwargs):
+ super().__init__(**kwargs)
+ self.connections_prefix = connections_prefix.rstrip(sep)
+ self.variables_prefix = variables_prefix.rstrip(sep)
+ self.vault_url = vault_url
+ self.sep = sep
+ self.kwargs = kwargs
+
+ @cached_property
+ def client(self):
+ """
+ Create a Azure Key Vault client.
+ """
+ credential = DefaultAzureCredential()
Review comment:
Would you mind to elaborate?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org