You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "amoghrajesh (via GitHub)" <gi...@apache.org> on 2025/05/19 14:54:06 UTC
[PR] Mask variable values in task logs only if the variable key is sensitive [airflow]
amoghrajesh opened a new pull request, #50775:
URL: https://github.com/apache/airflow/pull/50775
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
For a dag like this:
```
from __future__ import annotations
from airflow.models.baseoperator import BaseOperator
from airflow import DAG
from airflow.sdk import Variable
class CustomOperator(BaseOperator):
def execute(self, context):
v = Variable.get(key="key1")
print("Got value", v)
with DAG("get-var-sdk", schedule=None, catchup=False) as dag:
CustomOperator(task_id="set_var")
```
We shouldn't mask the value of the variable in logs unless the key is sensitive. As of now, we always mask it. This means that irrespective of the value, it will always be masked. This will block users from printing variable values in their tasks.
After changes:

<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mask variable values in task logs only if the variable key is sensitive [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #50775:
URL: https://github.com/apache/airflow/pull/50775#discussion_r2099949340
##########
task-sdk/tests/task_sdk/execution_time/test_supervisor.py:
##########
@@ -1524,6 +1522,10 @@ def test_handle_requests(
next(generator)
msg = message.model_dump_json().encode() + b"\n"
generator.send(msg)
+
+ if isinstance(message, GetVariable):
Review Comment:
Yeah i thought so, let me just add a param for this to check if it needs masking?
That way i will extend it to cover GetConnection too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mask variable values in task logs only if the variable key is sensitive [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #50775:
URL: https://github.com/apache/airflow/pull/50775#discussion_r2097345574
##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -1034,7 +1034,7 @@ def _handle_request(self, msg: ToSupervisor, log: FilteringBoundLogger):
var = self.client.variables.get(msg.key)
if isinstance(var, VariableResponse):
if var.value:
- mask_secret(var.value)
+ mask_secret(var.value, var.key)
Review Comment:
Yeah just pushed one through https://github.com/apache/airflow/pull/50775/commits/5ec9a8d6d1ccf05c8156eefab8a661a6ab4cca92.
Wondering if i should have it be more generic. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mask variable values in task logs only if the variable key is sensitive [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #50775:
URL: https://github.com/apache/airflow/pull/50775#discussion_r2097674386
##########
task-sdk/tests/task_sdk/execution_time/test_supervisor.py:
##########
@@ -1524,6 +1526,11 @@ def test_handle_requests(
next(generator)
msg = message.model_dump_json().encode() + b"\n"
generator.send(msg)
+
+ if isinstance(message, GetVariable):
+ mock_mask_secret.assert_called_once()
+ mock_mask_secret.assert_called_with("test_value", "test_key")
Review Comment:
```suggestion
mock_mask_secret.assert_called_with("test_value", "test_key")
```
We don't need both asserts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mask variable values in task logs only if the variable key is sensitive [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #50775:
URL: https://github.com/apache/airflow/pull/50775#discussion_r2099570377
##########
task-sdk/tests/task_sdk/execution_time/test_supervisor.py:
##########
@@ -1490,6 +1491,7 @@ def watched_subprocess(self, mocker):
def test_handle_requests(
self,
mock_secrets_masker,
+ mock_mask_secret,
Review Comment:
Yeah let me try and simplify it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mask variable values in task logs only if the variable key is sensitive [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh merged PR #50775:
URL: https://github.com/apache/airflow/pull/50775
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mask variable values in task logs only if the variable key is sensitive [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #50775:
URL: https://github.com/apache/airflow/pull/50775#discussion_r2099947658
##########
task-sdk/tests/task_sdk/execution_time/test_supervisor.py:
##########
@@ -1524,6 +1522,10 @@ def test_handle_requests(
next(generator)
msg = message.model_dump_json().encode() + b"\n"
generator.send(msg)
+
+ if isinstance(message, GetVariable):
Review Comment:
I don't like special casing this. If we need to do that, better to have a separate test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mask variable values in task logs only if the variable key is sensitive [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50775:
URL: https://github.com/apache/airflow/pull/50775#issuecomment-2898947880
### Backport successfully created: v3-0-test
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>✅</td>
<td>v3-0-test</td>
<td><a href="https://github.com/apache/airflow/pull/50927"><img src="https://img.shields.io/badge/PR-50927-blue" alt="PR Link"></a></td>
</tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mask variable values in task logs only if the variable key is sensitive [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #50775:
URL: https://github.com/apache/airflow/pull/50775#discussion_r2099569966
##########
task-sdk/tests/task_sdk/execution_time/test_supervisor.py:
##########
@@ -1524,6 +1526,11 @@ def test_handle_requests(
next(generator)
msg = message.model_dump_json().encode() + b"\n"
generator.send(msg)
+
+ if isinstance(message, GetVariable):
+ mock_mask_secret.assert_called_once()
+ mock_mask_secret.assert_called_with("test_value", "test_key")
Review Comment:
Yeah let me simplify this further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mask variable values in task logs only if the variable key is sensitive [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #50775:
URL: https://github.com/apache/airflow/pull/50775#discussion_r2096056860
##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -1034,7 +1034,7 @@ def _handle_request(self, msg: ToSupervisor, log: FilteringBoundLogger):
var = self.client.variables.get(msg.key)
if isinstance(var, VariableResponse):
if var.value:
- mask_secret(var.value)
+ mask_secret(var.value, var.key)
Review Comment:
Tests please!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mask variable values in task logs only if the variable key is sensitive [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #50775:
URL: https://github.com/apache/airflow/pull/50775#discussion_r2099576553
##########
task-sdk/tests/task_sdk/execution_time/test_supervisor.py:
##########
@@ -1490,6 +1491,7 @@ def watched_subprocess(self, mocker):
def test_handle_requests(
self,
mock_secrets_masker,
+ mock_mask_secret,
Review Comment:
I got rid of `mock_secrets_masker` as the `mock_mask_secret` will anyways initialise a secrets masker object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mask variable values in task logs only if the variable key is sensitive [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #50775:
URL: https://github.com/apache/airflow/pull/50775#discussion_r2100362422
##########
task-sdk/tests/task_sdk/execution_time/test_supervisor.py:
##########
@@ -1406,6 +1452,7 @@ def watched_subprocess(self, mocker):
(TI_ID, 1),
{},
TaskRescheduleStartDate(start_date=timezone.parse("2024-10-31T12:00:00Z")),
+ [],
Review Comment:
Handled in 1c0c15b3faa1e3e38384eca88c9f2e4c19a8d7ec
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mask variable values in task logs only if the variable key is sensitive [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #50775:
URL: https://github.com/apache/airflow/pull/50775#issuecomment-2897772905
@kaxil i just parameterised the test if something needs masking, wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mask variable values in task logs only if the variable key is sensitive [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #50775:
URL: https://github.com/apache/airflow/pull/50775#discussion_r2097569399
##########
task-sdk/tests/task_sdk/execution_time/test_supervisor.py:
##########
@@ -1524,6 +1526,11 @@ def test_handle_requests(
next(generator)
msg = message.model_dump_json().encode() + b"\n"
generator.send(msg)
+
+ if isinstance(message, GetVariable):
+ mock_mask_secret.assert_called_once()
+ mock_mask_secret.assert_called_with("test_value", "test_key")
Review Comment:
This has a change to be sliently never checked if we accidentally change the GetVariable fixture. It’s probably better to check these in a separate test.
##########
task-sdk/tests/task_sdk/execution_time/test_supervisor.py:
##########
@@ -1524,6 +1526,11 @@ def test_handle_requests(
next(generator)
msg = message.model_dump_json().encode() + b"\n"
generator.send(msg)
+
+ if isinstance(message, GetVariable):
+ mock_mask_secret.assert_called_once()
+ mock_mask_secret.assert_called_with("test_value", "test_key")
Review Comment:
This has a chance to be sliently never checked if we accidentally change the GetVariable fixture. It’s probably better to check these in a separate test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mask variable values in task logs only if the variable key is sensitive [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #50775:
URL: https://github.com/apache/airflow/pull/50775#discussion_r2097677082
##########
task-sdk/tests/task_sdk/execution_time/test_supervisor.py:
##########
@@ -1490,6 +1491,7 @@ def watched_subprocess(self, mocker):
def test_handle_requests(
self,
mock_secrets_masker,
+ mock_mask_secret,
Review Comment:
Why do we need both mock_secrets_masker and mock_mask_secret? That feels like we over mocking -- I don't think we should need both. Could we assert something on `mock_secrets_masker.return_value` without having to add an extra mock? Or can we get rid of one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mask variable values in task logs only if the variable key is sensitive [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #50775:
URL: https://github.com/apache/airflow/pull/50775#discussion_r2100179169
##########
task-sdk/tests/task_sdk/execution_time/test_supervisor.py:
##########
@@ -1406,6 +1452,7 @@ def watched_subprocess(self, mocker):
(TI_ID, 1),
{},
TaskRescheduleStartDate(start_date=timezone.parse("2024-10-31T12:00:00Z")),
+ [],
Review Comment:
Nit: Maybe all these should be none instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mask variable values in task logs only if the variable key is sensitive [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #50775:
URL: https://github.com/apache/airflow/pull/50775#discussion_r2099949340
##########
task-sdk/tests/task_sdk/execution_time/test_supervisor.py:
##########
@@ -1524,6 +1522,10 @@ def test_handle_requests(
next(generator)
msg = message.model_dump_json().encode() + b"\n"
generator.send(msg)
+
+ if isinstance(message, GetVariable):
Review Comment:
Yeah i thought so, let me just add a param for this to check if it needs masking?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mask variable values in task logs only if the variable key is sensitive [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #50775:
URL: https://github.com/apache/airflow/pull/50775#discussion_r2100353098
##########
task-sdk/tests/task_sdk/execution_time/test_supervisor.py:
##########
@@ -1406,6 +1452,7 @@ def watched_subprocess(self, mocker):
(TI_ID, 1),
{},
TaskRescheduleStartDate(start_date=timezone.parse("2024-10-31T12:00:00Z")),
+ [],
Review Comment:
Yeah not an issue at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Mask variable values in task logs only if the variable key is sensitive [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #50775:
URL: https://github.com/apache/airflow/pull/50775#discussion_r2097646739
##########
task-sdk/tests/task_sdk/execution_time/test_supervisor.py:
##########
@@ -1524,6 +1526,11 @@ def test_handle_requests(
next(generator)
msg = message.model_dump_json().encode() + b"\n"
generator.send(msg)
+
+ if isinstance(message, GetVariable):
+ mock_mask_secret.assert_called_once()
+ mock_mask_secret.assert_called_with("test_value", "test_key")
Review Comment:
@uranusjr GetVariable is a class not a fixture from comms.py, if we bulk update it, this test gets covered too. Although i guess we can handle it more generally?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org