You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "galafis (via GitHub)" <gi...@apache.org> on 2026/03/16 16:33:07 UTC
[PR] Mask sensitive pipeline variables in Beam triggers [airflow]
galafis opened a new pull request, #63738:
URL: https://github.com/apache/airflow/pull/63738
## What this fixes
Closes #54981
When `BeamRunPythonPipelineOperator` runs in **non-deferred** mode, Airflow's `SecretsMasker` picks up sensitive values (passwords, API tokens, etc.) from the standard logging pipeline and redacts them automatically. However, in **deferred** mode the operator hands off to `BeamPythonPipelineTrigger`, which stores the raw `variables` dict. When the triggerer process logs the trigger's state (via `__repr__` → `serialize()`), those sensitive values show up unmasked in the logs.
## How it works
This PR adds a small helper `_register_sensitive_variables()` that scans the pipeline `variables` dict for keys commonly associated with credentials (`password`, `token`, `api_key`, `secret`, etc.) and registers their values with `mask_secret()` from Airflow's `SecretsMasker`. This is called in `__init__` of both `BeamPythonPipelineTrigger` and `BeamJavaPipelineTrigger`, so the masker knows about those values before any logging happens.
The approach is consistent with how other parts of Airflow handle secret masking - registering values so the global `SecretsMasker` filter can redact them wherever they appear in log output.
## Changes
- Added `_SENSITIVE_VARIABLE_KEYS` frozenset with common sensitive key patterns
- Added `_register_sensitive_variables()` helper function
- Called the helper in `BeamPythonPipelineTrigger.__init__()` and `BeamJavaPipelineTrigger.__init__()`
- Added import for `mask_secret` from `airflow.utils.log.secrets_masker`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mask sensitive pipeline variables in Beam triggers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #63738:
URL: https://github.com/apache/airflow/pull/63738#issuecomment-4179322362
@galafis This PR has been converted to **draft** because it does not yet meet our [Pull Request quality criteria](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-quality-criteria).
**Issues found:**
- :x: **Pre-commit / static checks**: Failing: CI image checks / Static checks. Run `prek run --from-ref main` locally to find and fix issues. See [Pre-commit / static checks docs](https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst).
- :x: **Provider tests**: Failing: provider distributions tests / Compat 2.11.1:P3.10:, provider distributions tests / Compat 3.0.6:P3.10:, provider distributions tests / Compat 3.1.8:P3.10:, Postgres tests: providers / DB-prov:Postgres:14:3.10:common.compat...google, MySQL tests: providers / DB-prov:MySQL:8.0:3.10:common.compat...google (+1 more). Run provider tests with `breeze run pytest <provider-test-path> -xvs`. See [Provider tests docs](https://github.com/apache/airflow/blob/main/contributing-docs/12_provider_distributions.rst).
- :warning: **Unresolved review comments**: This PR has 1 unresolved review thread from maintainers: **@eladkal** (MEMBER): 1 unresolved thread. Please review and resolve all inline review comments before requesting another review. You can resolve a conversation by clicking 'Resolve conversation' on each thread after addressing the feedback. See [pull request guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst).
> **Note:** Your branch is **311 commits behind `main`**. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.
**What to do next:**
- The comment informs you what you need to do.
- Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
- There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates.
- Maintainers will then proceed with a normal review.
Converting a PR to draft is **not** a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the [Airflow Slack](https://s.apache.org/airflow-slack).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mask sensitive pipeline variables in Beam triggers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #63738: Mask sensitive pipeline variables in Beam triggers
URL: https://github.com/apache/airflow/pull/63738
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mask sensitive pipeline variables in Beam triggers [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #63738:
URL: https://github.com/apache/airflow/pull/63738#discussion_r2976342798
##########
providers/apache/beam/src/airflow/providers/apache/beam/triggers/beam.py:
##########
@@ -23,6 +23,37 @@
from airflow.providers.apache.beam.hooks.beam import BeamAsyncHook
from airflow.triggers.base import BaseTrigger, TriggerEvent
+from airflow.utils.log.secrets_masker import mask_secret
+
+# Keys in pipeline variables that may contain sensitive values and should
+# be masked in log output. This list covers the most common Beam pipeline
+# options that carry credentials.
+_SENSITIVE_VARIABLE_KEYS = frozenset(
+ {
+ "password",
+ "secret",
+ "token",
+ "api_key",
+ "apikey",
+ "api_token",
+ "access_token",
+ "refresh_token",
+ "credentials",
+ "private_key",
+ "service_account_key",
+ }
+)
+
+
+def _register_sensitive_variables(variables: dict) -> None:
+ """Register values of sensitive keys with Airflow's SecretsMasker.
+
+ This ensures that any log line that contains these values will have them
+ automatically replaced with ``***``.
+ """
+ for key, value in variables.items():
+ if value and any(sensitive in key.lower() for sensitive in _SENSITIVE_VARIABLE_KEYS):
+ mask_secret(value)
Review Comment:
Before we make the fix beam specific we need to understand if this happens only in beam - I have doubts that it does.
Can you offer confirmation for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mask sensitive pipeline variables in Beam triggers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #63738:
URL: https://github.com/apache/airflow/pull/63738#issuecomment-4296503195
@galafis This draft PR has been inactive for 20 days since the last triage comment and no response from the author. Closing to keep the queue clean.
You are welcome to reopen this PR when you resume work, or to open a new one addressing the issues previously raised. There is no rush — take your time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org