You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "amoghrajesh (via GitHub)" <gi...@apache.org> on 2025/03/24 09:49:49 UTC
[PR] Mask task logs generated by task SDK [airflow]
amoghrajesh opened a new pull request, #48186:
URL: https://github.com/apache/airflow/pull/48186
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
We need the ability to mask secrets in logs generated by tasks as they can log stuff in the logs which we do not want to display in the task logs.
Example of such a badly written DAG:
```
from airflow import DAG
from airflow.providers.standard.operators.python import PythonOperator
from airflow.sdk import Connection
def print_sensitive():
c = Connection.get("teradata_default")
print("Connection is", c.password)
with DAG(
dag_id="print_sensitive_data",
schedule=None,
) as dag:
print_sensitive_task = PythonOperator(
task_id="print_sensitive_task",
python_callable=print_sensitive,
)
```
- I am registering a new processor in the task sdk's configure_logging that will handle the log masking in the dict
- Decided to not redact every extra field. We need to worry about "event" field in event_dict as its generated by user tasks. Example of a event_dict:
```
{'chan': 'stdout', 'event': 'Connection is password', 'timestamp': '2025-03-24T08:49:41.302433Z', 'level': 'info', 'logger': 'task'}
```
- Added the registration of sensitive values in supervise() as this will ensure that every time a worker starts, it will register the masking.
- Tested with LocalExecutor, CeleryExecutor.
Before changes, logs:
<img width="1636" alt="image" src="https://github.com/user-attachments/assets/d300cb7d-f023-450f-93a4-942b0776960b" />
After changes, logs:
<img width="1636" alt="image" src="https://github.com/user-attachments/assets/98e63e88-645a-41c7-8bd5-ae1513308d44" />
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implementing masking for task SDK logs [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #48186:
URL: https://github.com/apache/airflow/pull/48186#discussion_r2010061778
##########
task-sdk/src/airflow/sdk/log.py:
##########
@@ -96,6 +96,13 @@ def redact_jwt(logger: Any, method_name: str, event_dict: EventDict) -> EventDic
return event_dict
+def mask_logs(logger: Any, method_name: str, event_dict: EventDict) -> EventDict:
+ from airflow.sdk.execution_time.secrets_masker import redact
+
+ event_dict["event"] = redact(event_dict["event"])
+ return event_dict
Review Comment:
We need to look at pretty much every field in `event_dict`:
If I do this:
```python
log.debug("Connecting", password=pass)`
```
it won't get redacted.
You can just pass the whole [dict to redact I think, and it will then also use the dict key names to redact, not just values](https://github.com/apache/airflow/blob/6a703dd1f795784e446742dddf412b0fe3b4e4a0/task-sdk/src/airflow/sdk/execution_time/secrets_masker.py#L238-L243).
However I think you will run into issues with this fn https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/secrets_masker.py#L192-L210 which expects a stdlib LogRecord object which we don't have anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implementing masking for task SDK logs [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #48186:
URL: https://github.com/apache/airflow/pull/48186#discussion_r2010124385
##########
task-sdk/src/airflow/sdk/log.py:
##########
@@ -96,6 +96,13 @@ def redact_jwt(logger: Any, method_name: str, event_dict: EventDict) -> EventDic
return event_dict
+def mask_logs(logger: Any, method_name: str, event_dict: EventDict) -> EventDict:
+ from airflow.sdk.execution_time.secrets_masker import redact
+
+ event_dict["event"] = redact(event_dict["event"])
+ return event_dict
Review Comment:
Thanks for the clarification, let me try that out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implementing masking for task SDK logs [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #48186:
URL: https://github.com/apache/airflow/pull/48186#discussion_r2011693114
##########
task-sdk/src/airflow/sdk/log.py:
##########
@@ -96,6 +96,13 @@ def redact_jwt(logger: Any, method_name: str, event_dict: EventDict) -> EventDic
return event_dict
+def mask_logs(logger: Any, method_name: str, event_dict: EventDict) -> EventDict:
+ from airflow.sdk.execution_time.secrets_masker import redact
+
+ event_dict["event"] = redact(event_dict["event"])
+ return event_dict
Review Comment:
Added a test and did the needful!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implementing masking for task SDK logs [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #48186:
URL: https://github.com/apache/airflow/pull/48186#discussion_r2010061778
##########
task-sdk/src/airflow/sdk/log.py:
##########
@@ -96,6 +96,13 @@ def redact_jwt(logger: Any, method_name: str, event_dict: EventDict) -> EventDic
return event_dict
+def mask_logs(logger: Any, method_name: str, event_dict: EventDict) -> EventDict:
+ from airflow.sdk.execution_time.secrets_masker import redact
+
+ event_dict["event"] = redact(event_dict["event"])
+ return event_dict
Review Comment:
We need to look at pretty much every field in `event_dict`:
If I do this:
```python
log.debug("Connecting", password=pass)`
```
it won't get redacted.
You can just pass the whole dict to redact I think, and it will then also use the dict key names to redact, not just values.
However I think you will run into issues with this fn https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/secrets_masker.py#L192-L210 which expects a stdlib LogRecord object which we don't have anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implementing masking for task SDK logs [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #48186:
URL: https://github.com/apache/airflow/pull/48186#discussion_r2009933664
##########
task-sdk/src/airflow/sdk/log.py:
##########
@@ -96,6 +96,13 @@ def redact_jwt(logger: Any, method_name: str, event_dict: EventDict) -> EventDic
return event_dict
+def mask_logs(logger: Any, method_name: str, event_dict: EventDict) -> EventDict:
+ from airflow.sdk.execution_time.secrets_masker import redact
+
+ event_dict["event"] = redact(event_dict["event"])
+ return event_dict
Review Comment:
Could you add a simple test case for it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implementing masking for task SDK logs [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh merged PR #48186:
URL: https://github.com/apache/airflow/pull/48186
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implementing masking for task SDK logs [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #48186:
URL: https://github.com/apache/airflow/pull/48186#issuecomment-2749165596
@ashb just tested with this DAG:
```
import logging
import structlog
from airflow import DAG
from airflow.providers.standard.operators.python import PythonOperator
from airflow.sdk import Connection
log = structlog.getLogger(__name__)
def print_sensitive():
c = Connection.get("teradata_default")
log.info("Connecting", password="password")
print("Connection is", c.password)
with DAG(
dag_id="print_sensitive_data",
schedule=None,
) as dag:
print_sensitive_task = PythonOperator(
task_id="print_sensitive_task",
python_callable=print_sensitive,
)
```
The redaction is happening as I expect it to happen:
The one qn is, are we ok with "source" gettin redacted?
```
[2025-03-24, 18:49:15] INFO - DAG bundles loaded: dags-folder source="***.dag_processing.bundles.manager.DagBundlesManager"
[2025-0[3](http://localhost:28080/dags/print_sensitive_data/runs/manual__2025-03-24T18:49:14.460932+00:00_9JfN0tZI/tasks/print_sensitive_task?try_number=1#3)-24, 18:49:15] INFO - Filling up the DagBag from /files/dags/dags/sensitive-dag.py source="***.models.dagbag.DagBag"
[2025-03-2[4](http://localhost:28080/dags/print_sensitive_data/runs/manual__2025-03-24T18:49:14.460932+00:00_9JfN0tZI/tasks/print_sensitive_task?try_number=1#4), 18:49:15] INFO - Connecting password="***" source="task"
[202[5](http://localhost:28080/dags/print_sensitive_data/runs/manual__2025-03-24T18:49:14.460932+00:00_9JfN0tZI/tasks/print_sensitive_task?try_number=1#5)-03-24, 18:49:15] INFO - Connection is *** chan="stdout" source="task"
[2025-03-24, 18:49:15] INFO - Done. Returned value was: None source="***.task.operators.***.providers.standard.operators.python.PythonOperator"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org