You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "PrithviBadiga (via GitHub)" <gi...@apache.org> on 2026/04/22 19:43:52 UTC
[PR] Add multi-team lookup to Azure Key Vault backend [airflow]
PrithviBadiga opened a new pull request, #65692:
URL: https://github.com/apache/airflow/pull/65692
Adds multi-team lookup support to `AzureKeyVaultBackend`.
Updates:
- look up team-scoped secrets first when `team_name` is provided
- fall back to the global secret when no team-scoped secret exists
- avoid resolving team-scoped identifiers as global secrets when `team_name` is not provided
- document the Azure team-scoped naming convention
Lookup behavior:
- team-scoped: `{prefix}-{team_name}-{secret_id}`
- global fallback: `{prefix}-{secret_id}`
Verification:
- `AIRFLOW_HOME=$(mktemp -d) PYTHONPATH=/Users/prith/Desktop/Codex/airflow-65682/airflow-core/src:/Users/prith/Desktop/Codex/airflow-65682/providers/microsoft/azure/src /Users/prith/Desktop/Codex/airflow/.venv/bin/python -m pytest /Users/prith/Desktop/Codex/airflow-65689-azure/providers/microsoft/azure/tests/unit/microsoft/azure/secrets/test_key_vault.py`
- `/Users/prith/Desktop/Codex/airflow/.venv/bin/python -m ruff check /Users/prith/Desktop/Codex/airflow-65689-azure/providers/microsoft/azure/src/airflow/providers/microsoft/azure/secrets/key_vault.py /Users/prith/Desktop/Codex/airflow-65689-azure/providers/microsoft/azure/tests/unit/microsoft/azure/secrets/test_key_vault.py`
- `/Users/prith/Desktop/Codex/airflow/.venv/bin/python -m ruff format --check /Users/prith/Desktop/Codex/airflow-65689-azure/providers/microsoft/azure/src/airflow/providers/microsoft/azure/secrets/key_vault.py /Users/prith/Desktop/Codex/airflow-65689-azure/providers/microsoft/azure/tests/unit/microsoft/azure/secrets/test_key_vault.py`
Part of: #65682
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add multi-team lookup to Azure Key Vault backend [airflow]
Posted by "PrithviBadiga (via GitHub)" <gi...@apache.org>.
PrithviBadiga commented on code in PR #65692:
URL: https://github.com/apache/airflow/pull/65692#discussion_r3133019862
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/secrets/key_vault.py:
##########
@@ -200,17 +207,41 @@ def build_path(path_prefix: str, secret_id: str, sep: str = "-") -> str:
path = f"{path_prefix}{sep}{secret_id}"
return path.replace("_", sep)
- def _get_secret(self, path_prefix: str, secret_id: str) -> str | None:
+ def _build_team_secret_name(self, path_prefix: str, team_name: str, secret_id: str) -> str:
+ """Build a team-scoped secret name using a dedicated separator before the secret id."""
+ team_prefix = self.build_path(path_prefix, team_name, self.sep)
+ normalized_secret_id = secret_id.replace("_", self.sep)
+ return f"{team_prefix}{self.TEAM_SEP}{normalized_secret_id}"
+
+ def _is_team_specific_accessed_as_global(self, secret_id: str, team_name: str | None = None) -> bool:
+ normalized_secret_id = secret_id.replace("_", self.sep)
+ team_pattern = rf"[^{re.escape(self.sep)}]+{re.escape(self.TEAM_SEP)}.+"
+ return team_name is None and bool(re.fullmatch(team_pattern, normalized_secret_id))
+
+ def _get_secret(self, path_prefix: str, secret_id: str, team_name: str | None = None) -> str | None:
"""
Get an Azure Key Vault secret value.
:param path_prefix: Prefix for the Path to get Secret
:param secret_id: Secret Key
"""
+ if team_name:
+ team_secret = self._get_secret_value(
+ path_prefix, self._build_team_secret_name("", team_name, secret_id)
Review Comment:
Good catch, yes. My bad!
I fixed the team lookup so it preserves `path_prefix`, and I added a regression test covering team lookup with a custom prefix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add multi-team lookup to Azure Key Vault backend [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck merged PR #65692:
URL: https://github.com/apache/airflow/pull/65692
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add multi-team lookup to Azure Key Vault backend [airflow]
Posted by "PrithviBadiga (via GitHub)" <gi...@apache.org>.
PrithviBadiga commented on code in PR #65692:
URL: https://github.com/apache/airflow/pull/65692#discussion_r3133115621
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/secrets/key_vault.py:
##########
@@ -200,13 +209,34 @@ def build_path(path_prefix: str, secret_id: str, sep: str = "-") -> str:
path = f"{path_prefix}{sep}{secret_id}"
return path.replace("_", sep)
- def _get_secret(self, path_prefix: str, secret_id: str) -> str | None:
+ def _build_team_secret_name(self, path_prefix: str, team_name: str, secret_id: str) -> str:
+ """Build a team-scoped secret name using a dedicated separator before the secret id."""
+ team_prefix = self.build_path(path_prefix, team_name, self.sep)
+ normalized_secret_id = secret_id.replace("_", self.sep)
Review Comment:
This is to preserve the existing Azure Key Vault naming behavior.
`build_path()` already normalizes `_` to the configured separator, so a secret id like `my_db` is stored/looked up as `my-db`. The team-scoped helper needs to apply the same normalization to the secret id, otherwise team-scoped lookups would behave differently from the existing global lookups for the same connection/variable name.
I can add a short clarifying comment if that would help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add multi-team lookup to Azure Key Vault backend [airflow]
Posted by "PrithviBadiga (via GitHub)" <gi...@apache.org>.
PrithviBadiga commented on code in PR #65692:
URL: https://github.com/apache/airflow/pull/65692#discussion_r3127416608
##########
providers/microsoft/azure/docs/secrets-backends/azure-key-vault.rst:
##########
@@ -67,6 +67,26 @@ Storing and Retrieving Variables
If you have set ``variables_prefix`` as ``airflow-variables``, then for an Variable key of ``hello``,
you would want to store your Variable at ``airflow-variables-hello``.
+Multi-team lookup
+"""""""""""""""""
+
+In multi-team mode, this backend looks for team-scoped secrets first and falls back to the global
+secret name when a team-scoped secret is not found.
+
+For connections:
+
+* Team-scoped: ``{connections_prefix}-{team_name}-{conn_id}``
+* Global fallback: ``{connections_prefix}-{conn_id}``
Review Comment:
Good point. I updated the Azure backend to use a dedicated separator between the team and secret id (`--`) so names like `marketing-secret` are no longer ambiguous, and I added matching detection to avoid resolving team-scoped secrets as global secrets.
Example:
- team-scoped: `airflow-connections-marketing--secret`
- global: `airflow-connections-secret`
I also updated the docs and tests accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add multi-team lookup to Azure Key Vault backend [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on code in PR #65692:
URL: https://github.com/apache/airflow/pull/65692#discussion_r3133192891
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/secrets/key_vault.py:
##########
@@ -200,13 +209,34 @@ def build_path(path_prefix: str, secret_id: str, sep: str = "-") -> str:
path = f"{path_prefix}{sep}{secret_id}"
return path.replace("_", sep)
- def _get_secret(self, path_prefix: str, secret_id: str) -> str | None:
+ def _build_team_secret_name(self, path_prefix: str, team_name: str, secret_id: str) -> str:
+ """Build a team-scoped secret name using a dedicated separator before the secret id."""
+ team_prefix = self.build_path(path_prefix, team_name, self.sep)
+ normalized_secret_id = secret_id.replace("_", self.sep)
Review Comment:
Oh I see, thanks for clarifying. No I think this is good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add multi-team lookup to Azure Key Vault backend [airflow]
Posted by "PrithviBadiga (via GitHub)" <gi...@apache.org>.
PrithviBadiga commented on code in PR #65692:
URL: https://github.com/apache/airflow/pull/65692#discussion_r3133015973
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/secrets/key_vault.py:
##########
@@ -200,17 +207,41 @@ def build_path(path_prefix: str, secret_id: str, sep: str = "-") -> str:
path = f"{path_prefix}{sep}{secret_id}"
return path.replace("_", sep)
- def _get_secret(self, path_prefix: str, secret_id: str) -> str | None:
+ def _build_team_secret_name(self, path_prefix: str, team_name: str, secret_id: str) -> str:
+ """Build a team-scoped secret name using a dedicated separator before the secret id."""
+ team_prefix = self.build_path(path_prefix, team_name, self.sep)
+ normalized_secret_id = secret_id.replace("_", self.sep)
+ return f"{team_prefix}{self.TEAM_SEP}{normalized_secret_id}"
+
+ def _is_team_specific_accessed_as_global(self, secret_id: str, team_name: str | None = None) -> bool:
+ normalized_secret_id = secret_id.replace("_", self.sep)
+ team_pattern = rf"[^{re.escape(self.sep)}]+{re.escape(self.TEAM_SEP)}.+"
+ return team_name is None and bool(re.fullmatch(team_pattern, normalized_secret_id))
Review Comment:
I simplified that guard to make it clearer.
The normalization is there because Azure secret names go through the same underscore-to-separator conversion as `build_path`, so I now reuse that normalization directly before checking whether the id already looks team-scoped. I also removed the extra `team_pattern` variable and replaced it with a simpler check based on the dedicated `--` separator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add multi-team lookup to Azure Key Vault backend [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on code in PR #65692:
URL: https://github.com/apache/airflow/pull/65692#discussion_r3126593359
##########
providers/microsoft/azure/docs/secrets-backends/azure-key-vault.rst:
##########
@@ -67,6 +67,26 @@ Storing and Retrieving Variables
If you have set ``variables_prefix`` as ``airflow-variables``, then for an Variable key of ``hello``,
you would want to store your Variable at ``airflow-variables-hello``.
+Multi-team lookup
+"""""""""""""""""
+
+In multi-team mode, this backend looks for team-scoped secrets first and falls back to the global
+secret name when a team-scoped secret is not found.
+
+For connections:
+
+* Team-scoped: ``{connections_prefix}-{team_name}-{conn_id}``
+* Global fallback: ``{connections_prefix}-{conn_id}``
Review Comment:
We had this issue in the environment variable secret backend: what if I try to look up the secret `marketing-secret`? It will try to look up the secret `secret` in the team `marketing` even though I do not belong to this team? See https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/secrets/environment_variables.py#L66
I would advise:
- Adding another separator like double dash `--` between the team and `conn_id`
- Detect the same we did in https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/secrets/environment_variables.py#L66 if a team secret is being access as a global secret
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add multi-team lookup to Azure Key Vault backend [airflow]
Posted by "PrithviBadiga (via GitHub)" <gi...@apache.org>.
PrithviBadiga commented on code in PR #65692:
URL: https://github.com/apache/airflow/pull/65692#discussion_r3133011730
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/secrets/key_vault.py:
##########
@@ -200,17 +207,41 @@ def build_path(path_prefix: str, secret_id: str, sep: str = "-") -> str:
path = f"{path_prefix}{sep}{secret_id}"
return path.replace("_", sep)
- def _get_secret(self, path_prefix: str, secret_id: str) -> str | None:
+ def _build_team_secret_name(self, path_prefix: str, team_name: str, secret_id: str) -> str:
+ """Build a team-scoped secret name using a dedicated separator before the secret id."""
+ team_prefix = self.build_path(path_prefix, team_name, self.sep)
+ normalized_secret_id = secret_id.replace("_", self.sep)
+ return f"{team_prefix}{self.TEAM_SEP}{normalized_secret_id}"
+
+ def _is_team_specific_accessed_as_global(self, secret_id: str, team_name: str | None = None) -> bool:
+ normalized_secret_id = secret_id.replace("_", self.sep)
+ team_pattern = rf"[^{re.escape(self.sep)}]+{re.escape(self.TEAM_SEP)}.+"
+ return team_name is None and bool(re.fullmatch(team_pattern, normalized_secret_id))
+
+ def _get_secret(self, path_prefix: str, secret_id: str, team_name: str | None = None) -> str | None:
"""
Get an Azure Key Vault secret value.
:param path_prefix: Prefix for the Path to get Secret
:param secret_id: Secret Key
"""
+ if team_name:
+ team_secret = self._get_secret_value(
+ path_prefix, self._build_team_secret_name("", team_name, secret_id)
+ )
+ if team_secret is not None:
+ return team_secret
+
+ return self._get_secret_value(path_prefix, secret_id)
+
+ def _get_secret_value(self, path_prefix: str, secret_id: str) -> str | None:
+ """Get an Azure Key Vault secret value for the given prefix and key."""
name = self.build_path(path_prefix, secret_id, self.sep)
try:
secret = self.client.get_secret(name=name)
return secret.value
except ResourceNotFoundError as ex:
self.log.debug("Secret %s not found: %s", name, ex)
return None
+
+ TEAM_SEP = "--"
Review Comment:
I moved `TEAM_SEP` to the top-level constants section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add multi-team lookup to Azure Key Vault backend [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on code in PR #65692:
URL: https://github.com/apache/airflow/pull/65692#discussion_r3133055611
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/secrets/key_vault.py:
##########
@@ -200,13 +209,34 @@ def build_path(path_prefix: str, secret_id: str, sep: str = "-") -> str:
path = f"{path_prefix}{sep}{secret_id}"
return path.replace("_", sep)
- def _get_secret(self, path_prefix: str, secret_id: str) -> str | None:
+ def _build_team_secret_name(self, path_prefix: str, team_name: str, secret_id: str) -> str:
+ """Build a team-scoped secret name using a dedicated separator before the secret id."""
+ team_prefix = self.build_path(path_prefix, team_name, self.sep)
+ normalized_secret_id = secret_id.replace("_", self.sep)
Review Comment:
Again, why do we need that? I do not quite understand why we want to replace underscore by dashes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add multi-team lookup to Azure Key Vault backend [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on code in PR #65692:
URL: https://github.com/apache/airflow/pull/65692#discussion_r3130888408
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/secrets/key_vault.py:
##########
@@ -200,17 +207,41 @@ def build_path(path_prefix: str, secret_id: str, sep: str = "-") -> str:
path = f"{path_prefix}{sep}{secret_id}"
return path.replace("_", sep)
- def _get_secret(self, path_prefix: str, secret_id: str) -> str | None:
+ def _build_team_secret_name(self, path_prefix: str, team_name: str, secret_id: str) -> str:
+ """Build a team-scoped secret name using a dedicated separator before the secret id."""
+ team_prefix = self.build_path(path_prefix, team_name, self.sep)
+ normalized_secret_id = secret_id.replace("_", self.sep)
+ return f"{team_prefix}{self.TEAM_SEP}{normalized_secret_id}"
+
+ def _is_team_specific_accessed_as_global(self, secret_id: str, team_name: str | None = None) -> bool:
+ normalized_secret_id = secret_id.replace("_", self.sep)
+ team_pattern = rf"[^{re.escape(self.sep)}]+{re.escape(self.TEAM_SEP)}.+"
+ return team_name is None and bool(re.fullmatch(team_pattern, normalized_secret_id))
Review Comment:
I am trying to understand this code, can you explain? Why are you replacing `-` by `_`? And what is `team_pattern`?
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/secrets/key_vault.py:
##########
@@ -200,17 +207,41 @@ def build_path(path_prefix: str, secret_id: str, sep: str = "-") -> str:
path = f"{path_prefix}{sep}{secret_id}"
return path.replace("_", sep)
- def _get_secret(self, path_prefix: str, secret_id: str) -> str | None:
+ def _build_team_secret_name(self, path_prefix: str, team_name: str, secret_id: str) -> str:
+ """Build a team-scoped secret name using a dedicated separator before the secret id."""
+ team_prefix = self.build_path(path_prefix, team_name, self.sep)
+ normalized_secret_id = secret_id.replace("_", self.sep)
+ return f"{team_prefix}{self.TEAM_SEP}{normalized_secret_id}"
+
+ def _is_team_specific_accessed_as_global(self, secret_id: str, team_name: str | None = None) -> bool:
+ normalized_secret_id = secret_id.replace("_", self.sep)
+ team_pattern = rf"[^{re.escape(self.sep)}]+{re.escape(self.TEAM_SEP)}.+"
+ return team_name is None and bool(re.fullmatch(team_pattern, normalized_secret_id))
+
+ def _get_secret(self, path_prefix: str, secret_id: str, team_name: str | None = None) -> str | None:
"""
Get an Azure Key Vault secret value.
:param path_prefix: Prefix for the Path to get Secret
:param secret_id: Secret Key
"""
+ if team_name:
+ team_secret = self._get_secret_value(
+ path_prefix, self._build_team_secret_name("", team_name, secret_id)
Review Comment:
Should not it be `path_prefix` instead of `""`?
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/secrets/key_vault.py:
##########
@@ -200,17 +207,41 @@ def build_path(path_prefix: str, secret_id: str, sep: str = "-") -> str:
path = f"{path_prefix}{sep}{secret_id}"
return path.replace("_", sep)
- def _get_secret(self, path_prefix: str, secret_id: str) -> str | None:
+ def _build_team_secret_name(self, path_prefix: str, team_name: str, secret_id: str) -> str:
+ """Build a team-scoped secret name using a dedicated separator before the secret id."""
+ team_prefix = self.build_path(path_prefix, team_name, self.sep)
+ normalized_secret_id = secret_id.replace("_", self.sep)
+ return f"{team_prefix}{self.TEAM_SEP}{normalized_secret_id}"
+
+ def _is_team_specific_accessed_as_global(self, secret_id: str, team_name: str | None = None) -> bool:
+ normalized_secret_id = secret_id.replace("_", self.sep)
+ team_pattern = rf"[^{re.escape(self.sep)}]+{re.escape(self.TEAM_SEP)}.+"
+ return team_name is None and bool(re.fullmatch(team_pattern, normalized_secret_id))
+
+ def _get_secret(self, path_prefix: str, secret_id: str, team_name: str | None = None) -> str | None:
"""
Get an Azure Key Vault secret value.
:param path_prefix: Prefix for the Path to get Secret
:param secret_id: Secret Key
"""
+ if team_name:
+ team_secret = self._get_secret_value(
+ path_prefix, self._build_team_secret_name("", team_name, secret_id)
+ )
+ if team_secret is not None:
+ return team_secret
+
+ return self._get_secret_value(path_prefix, secret_id)
+
+ def _get_secret_value(self, path_prefix: str, secret_id: str) -> str | None:
+ """Get an Azure Key Vault secret value for the given prefix and key."""
name = self.build_path(path_prefix, secret_id, self.sep)
try:
secret = self.client.get_secret(name=name)
return secret.value
except ResourceNotFoundError as ex:
self.log.debug("Secret %s not found: %s", name, ex)
return None
+
+ TEAM_SEP = "--"
Review Comment:
At the top of the file please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org