You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Subham-KRLX (via GitHub)" <gi...@apache.org> on 2026/04/29 11:11:07 UTC
[PR] Fix AirflowSDKConfigParser missing mask_secrets method [airflow]
Subham-KRLX opened a new pull request, #66077:
URL: https://github.com/apache/airflow/pull/66077
AirflowSDKConfigParser was missing the mask_secrets() method which caused an AttributeError on startup whenever settings.initialize() called conf.mask_secrets() (e.g. running airflow db migrate).
The method existed only on the core AirflowConfigParser and was never added to the SDK subclass. Added mask_secrets() to AirflowSDKConfigParser using SDK-native imports, matching the same logic as the core implementation.
closes: #66074
Was generative AI tooling used to co-author this PR?
Yes — Claude (For Pr description)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix AirflowSDKConfigParser missing mask_secrets method [airflow]
Posted by "stephen-bracken (via GitHub)" <gi...@apache.org>.
stephen-bracken commented on PR #66077:
URL: https://github.com/apache/airflow/pull/66077#issuecomment-4343796814
I think this needs to be added to the superclass in `airflow.shared.configuration.parser.AirflowConfigParser`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix AirflowSDKConfigParser missing mask_secrets method [airflow]
Posted by "Subham-KRLX (via GitHub)" <gi...@apache.org>.
Subham-KRLX commented on PR #66077:
URL: https://github.com/apache/airflow/pull/66077#issuecomment-4345341934
Thanks for tracking that down the compat provider intentionally routes conf to airflow.sdk.configuration on Airflow 3 so AirflowSDKConfigParser can legitimately end up as the active conf this fix ensures it works correctly in that scenario. Whether the compat routing should be revisited is a separate call for maintainers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix AirflowSDKConfigParser missing mask_secrets method [airflow]
Posted by "Subham-KRLX (via GitHub)" <gi...@apache.org>.
Subham-KRLX commented on PR #66077:
URL: https://github.com/apache/airflow/pull/66077#issuecomment-4343826112
> I think this needs to be added to the superclass in `airflow.shared.configuration.parser.AirflowConfigParser`?
The shared base has zero secrets masker imports by design it's a pure parsing library Core's AirflowConfigParser defines mask_secrets() at the core layer for the same reason each subclass owns its masking using its own secrets masker coupling the shared base to secrets masking would be the wrong direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix AirflowSDKConfigParser missing mask_secrets method [airflow]
Posted by "stephen-bracken (via GitHub)" <gi...@apache.org>.
stephen-bracken commented on PR #66077:
URL: https://github.com/apache/airflow/pull/66077#issuecomment-4345306612
I found the cause, I was importing `conf` from `airflow.providers.common.compat.sdk` in my `airflow_local_settings.py`, which was defaulting to the sdk config parser
https://github.com/apache/airflow/issues/66074#issuecomment-4344384860
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix AirflowSDKConfigParser missing mask_secrets method [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #66077:
URL: https://github.com/apache/airflow/pull/66077#issuecomment-4642230781
### Backport successfully created: v3-2-test
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>✅</td>
<td>v3-2-test</td>
<td><a href="https://github.com/apache/airflow/pull/68158"><img src="https://img.shields.io/badge/PR-68158-blue" alt="PR Link"></a></td>
</tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix AirflowSDKConfigParser missing mask_secrets method [airflow]
Posted by "stephen-bracken (via GitHub)" <gi...@apache.org>.
stephen-bracken commented on PR #66077:
URL: https://github.com/apache/airflow/pull/66077#issuecomment-4344231455
one further point - the error from the issue occurs when importing `airflow.configuration.conf` from `airflow.settings`, which does implement the `mask_secrets()` method. I think we need to understand why the sub class isn't instantiated there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix AirflowSDKConfigParser missing mask_secrets method [airflow]
Posted by "Subham-KRLX (via GitHub)" <gi...@apache.org>.
Subham-KRLX commented on PR #66077:
URL: https://github.com/apache/airflow/pull/66077#issuecomment-4530802780
@vatsrahul1001
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix AirflowSDKConfigParser missing mask_secrets method [airflow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #66077:
URL: https://github.com/apache/airflow/pull/66077#discussion_r3368119817
##########
task-sdk/src/airflow/sdk/configuration.py:
##########
@@ -161,6 +161,35 @@ def __init__(
if default_config is not None:
self._update_defaults_from_string(default_config)
+ def mask_secrets(self) -> None:
+ """Mask sensitive config values in the secrets masker."""
+ from airflow.sdk._shared.configuration.exceptions import AirflowConfigException
+ from airflow.sdk._shared.configuration.parser import _build_kwarg_env_prefix, _collect_kwarg_env_vars
+ from airflow.sdk._shared.secrets_masker import mask_secret
+
+ for section, key in self.sensitive_config_values:
+ try:
+ with self.suppress_future_warnings():
+ value = self.get(section, key, suppress_warnings=True)
+ except AirflowConfigException:
+ log.debug(
+ "Could not retrieve value from section %s, for key %s. Skipping redaction of this conf.",
+ section,
+ key,
+ )
+ continue
+ mask_secret(value)
+
+ # Mask per-key backend kwarg env vars (AIRFLOW__SECRETS__BACKEND_KWARG__* etc.).
+ # These are not in sensitive_config_values but may contain sensitive values.
+ for _section, _kwargs_key in [
+ ("secrets", "backend_kwargs"),
+ ("workers", "secrets_backend_kwargs"),
+ ]:
+ _prefix = _build_kwarg_env_prefix(_section, _kwargs_key)
+ for _value in _collect_kwarg_env_vars(_prefix).values():
+ mask_secret(_value)
Review Comment:
`mask_secrets()` currently only registers values with the Task SDK secrets masker (`airflow.sdk._shared.secrets_masker`). In the reported failure mode (user’s `airflow_local_settings` overwriting `airflow.settings.conf` with SDK `conf`), `settings.initialize()` still configures *both* the core and SDK maskers, and core logging continues to use the core masker. If secrets are not also registered with `airflow._shared.secrets_masker`, sensitive config values can leak into core logs even though `conf.mask_secrets()` was called.
##########
task-sdk/tests/task_sdk/test_configuration.py:
##########
@@ -184,3 +184,25 @@ def test_expands_env_var_in_airflow_home_fallback(self):
env = {"AIRFLOW_HOME": "$CUSTOM_DIR/airflow", "CUSTOM_DIR": "/resolved"}
with mock.patch.dict("os.environ", env, clear=True):
assert get_airflow_config() == "/resolved/airflow/airflow.cfg"
+
+
+class TestAirflowSDKConfigParser:
+ @mock.patch("airflow.sdk._shared.secrets_masker.mask_secret")
+ def test_mask_secrets(self, mock_mask_secret):
+ from airflow.sdk.configuration import AirflowSDKConfigParser
+
+ parser = AirflowSDKConfigParser()
+
+ # Set a sensitive value
+ parser.sensitive_config_values.add(("test_section", "secret_key"))
+ if not parser.has_section("test_section"):
+ parser.add_section("test_section")
+ parser.set("test_section", "secret_key", "super_secret_value")
+
+ with mock.patch.dict(
+ "os.environ", {"AIRFLOW__SECRETS__BACKEND_KWARG__SOME_KEY": "secret_kwarg_value"}
+ ):
+ parser.mask_secrets()
+
+ mock_mask_secret.assert_any_call("super_secret_value")
+ mock_mask_secret.assert_any_call("secret_kwarg_value")
Review Comment:
The new test only asserts that the SDK secrets masker is called. If `AirflowSDKConfigParser.mask_secrets()` is meant to mirror the core implementation and prevent secrets from leaking in *core* log redaction too, this test should also verify that `airflow._shared.secrets_masker.mask_secret` is invoked when available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix AirflowSDKConfigParser missing mask_secrets method [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #66077:
URL: https://github.com/apache/airflow/pull/66077#issuecomment-4638265373
Pytest missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix AirflowSDKConfigParser missing mask_secrets method [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl merged PR #66077:
URL: https://github.com/apache/airflow/pull/66077
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix AirflowSDKConfigParser missing mask_secrets method [airflow]
Posted by "Subham-KRLX (via GitHub)" <gi...@apache.org>.
Subham-KRLX commented on PR #66077:
URL: https://github.com/apache/airflow/pull/66077#issuecomment-4345282289
> one further point - the error from the issue occurs when importing `airflow.configuration.conf` from `airflow.settings`, which does implement the `mask_secrets()` method. I think we need to understand why the sub class isn't instantiated there
Good point settings.py does import conf from airflow.configuration which already has mask_secrets() so the real question is why AirflowSDKConfigParser ends up as conf in the reporter's setup my fix makes the SDK parser safe to call regardless but the root cause of why the wrong class is being used as conf is worth digging into further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org