You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "sarth-akvaish (via GitHub)" <gi...@apache.org> on 2026/01/18 07:06:40 UTC
[PR] Bigquery SQL Renderering Issue [airflow]
sarth-akvaish opened a new pull request, #60739:
URL: https://github.com/apache/airflow/pull/60739
This PR resolves the issue of not rendering SQL (other also like bash command) in the Rendered Templates tab in the specific Task.
Fixes : https://github.com/apache/airflow/issues/60589
### **What is happening currently ?**
Rendered Template fields are generated inside the task runner after jinja rendering and then sent to the API server to be stored in the RTIF table.
The task runner only serialized the top-level template fields defined on the operator(for ex - configuration)
Nested renderer paths (such as configuration.query.query) were not extracted or sent.
As a result The RTIF model and UI were functioning correctly, but the required data never reached them.
### **What changed made**
The task runner logic si updated to corrrectly capture rendered template values after rendering and senfd them to the API server.
Becuase the UI is data driven, it immediately began displaying the correct rendered templates with proper title and syntax, without any changes in teh UI code.
Image after fixing this issue
<img width="1297" height="694" alt="Screenshot from 2026-01-18 12-23-18" src="https://github.com/user-attachments/assets/702db139-6d4a-4eb0-a0d8-470edd9f08f0" />
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull request.
Write a good git commit message following this guide: http://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the documentation.
Feel free to ping (in general) for the review if you do not see reaction for a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Bugfix: SQL not rendered in Rendered Templates view [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60739:
URL: https://github.com/apache/airflow/pull/60739#issuecomment-3943840679
### Backport successfully created: v3-1-test
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>✅</td>
<td>v3-1-test</td>
<td><a href="https://github.com/apache/airflow/pull/62348"><img src="https://img.shields.io/badge/PR-62348-blue" alt="PR Link"></a></td>
</tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Bugfix: SQL not rendered in Rendered Templates view [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh merged PR #60739:
URL: https://github.com/apache/airflow/pull/60739
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Bugfix: SQL not rendered in Rendered Templates view [airflow]
Posted by "sarth-akvaish (via GitHub)" <gi...@apache.org>.
sarth-akvaish commented on code in PR #60739:
URL: https://github.com/apache/airflow/pull/60739#discussion_r2798107584
##########
task-sdk/tests/task_sdk/execution_time/test_task_runner.py:
##########
@@ -2702,6 +2702,119 @@ def execute(self, context):
== '[{"name": "var1", "value": "This is a test phrase.", "value_from": null}, {"name": "var2", "value": "***", "value_from": null}, {"name": "var3", "value": "***", "value_from": null}]'
)
+ def test_rendered_fields_are_redacted_before_validation(self, create_runtime_ti, mock_supervisor_comms):
+ """
+ Ensure that redact() is applied before JsonValue validation
+ and the redacted value is what is sent in SetRenderedFields.
+ """
+ from airflow.sdk.execution_time.comms import SetRenderedFields
+ from airflow.sdk.execution_time.task_runner import run
+
+ secret_value = "super_secret_password"
+
+ class CustomOperator(BaseOperator):
+ template_fields = ("password",)
+
+ def __init__(self, password, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+ self.password = password
+
+ def execute(self, context):
+ pass
+
+ task = CustomOperator(
+ task_id="redact_test",
+ password=secret_value,
+ )
+
+ runtime_ti = create_runtime_ti(task=task, dag_id="redact_test_dag")
+
+ with mock.patch(
+ "airflow.sdk._shared.secrets_masker.redact",
+ return_value="***REDACTED***",
+ ) as mock_redact:
+ run(runtime_ti, context=runtime_ti.get_template_context(), log=mock.MagicMock())
+
+ mock_redact.assert_called_once_with(secret_value, "password")
+
+ assert any(
+ isinstance(call.kwargs.get("msg"), SetRenderedFields)
+ and call.kwargs["msg"].rendered_fields["password"] == "***REDACTED***"
+ for call in mock_supervisor_comms.send.mock_calls
+ )
Review Comment:
Ok, will remove these tests and update the PR. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Bugfix: SQL not rendered in Rendered Templates view [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #60739:
URL: https://github.com/apache/airflow/pull/60739#discussion_r2798046615
##########
task-sdk/tests/task_sdk/execution_time/test_task_runner.py:
##########
@@ -2702,6 +2702,119 @@ def execute(self, context):
== '[{"name": "var1", "value": "This is a test phrase.", "value_from": null}, {"name": "var2", "value": "***", "value_from": null}, {"name": "var3", "value": "***", "value_from": null}]'
)
+ def test_rendered_fields_are_redacted_before_validation(self, create_runtime_ti, mock_supervisor_comms):
+ """
+ Ensure that redact() is applied before JsonValue validation
+ and the redacted value is what is sent in SetRenderedFields.
+ """
+ from airflow.sdk.execution_time.comms import SetRenderedFields
+ from airflow.sdk.execution_time.task_runner import run
+
+ secret_value = "super_secret_password"
+
+ class CustomOperator(BaseOperator):
+ template_fields = ("password",)
+
+ def __init__(self, password, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+ self.password = password
+
+ def execute(self, context):
+ pass
+
+ task = CustomOperator(
+ task_id="redact_test",
+ password=secret_value,
+ )
+
+ runtime_ti = create_runtime_ti(task=task, dag_id="redact_test_dag")
+
+ with mock.patch(
+ "airflow.sdk._shared.secrets_masker.redact",
+ return_value="***REDACTED***",
+ ) as mock_redact:
+ run(runtime_ti, context=runtime_ti.get_template_context(), log=mock.MagicMock())
+
+ mock_redact.assert_called_once_with(secret_value, "password")
+
+ assert any(
+ isinstance(call.kwargs.get("msg"), SetRenderedFields)
+ and call.kwargs["msg"].rendered_fields["password"] == "***REDACTED***"
+ for call in mock_supervisor_comms.send.mock_calls
+ )
Review Comment:
I don't think we need this test, we cover the two basic tests below. Order of operations is ok to not test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Bigquery SQL Renderering Issue [airflow]
Posted by "sarth-akvaish (via GitHub)" <gi...@apache.org>.
sarth-akvaish commented on PR #60739:
URL: https://github.com/apache/airflow/pull/60739#issuecomment-3874549984
> Change looks about okay, but please add some unit tests covering this
Sure, will update the PR. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Bugfix: SQL not rendered in Rendered Templates view [airflow]
Posted by "sarth-akvaish (via GitHub)" <gi...@apache.org>.
sarth-akvaish commented on PR #60739:
URL: https://github.com/apache/airflow/pull/60739#issuecomment-3897233072
Updated the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Bigquery SQL Renderering Issue [airflow]
Posted by "nathadfield (via GitHub)" <gi...@apache.org>.
nathadfield commented on PR #60739:
URL: https://github.com/apache/airflow/pull/60739#issuecomment-3872352941
@sarth-akvaish I noticed that no one has given this a review yet. @ashb, @amoghrajesh and @kaxil if one of you gets a spare moment to take a look, we'd be most appreciative.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Bigquery SQL Renderering Issue [airflow]
Posted by "nathadfield (via GitHub)" <gi...@apache.org>.
nathadfield commented on PR #60739:
URL: https://github.com/apache/airflow/pull/60739#issuecomment-3885097574
@sarth-akvaish Looks like there are some conflicts that need resolving too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Bugfix: SQL not rendered in Rendered Templates view [airflow]
Posted by "nathadfield (via GitHub)" <gi...@apache.org>.
nathadfield commented on PR #60739:
URL: https://github.com/apache/airflow/pull/60739#issuecomment-3943412120
Is someone with merge powers prepared to press the button on this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Bugfix: SQL not rendered in Rendered Templates view [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #60739:
URL: https://github.com/apache/airflow/pull/60739#issuecomment-3943820808
@nathadfield done! Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Bigquery SQL Renderering Issue [airflow]
Posted by "sarth-akvaish (via GitHub)" <gi...@apache.org>.
sarth-akvaish commented on PR #60739:
URL: https://github.com/apache/airflow/pull/60739#issuecomment-3885047977
Hi @ashb, I've added the tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Bugfix: SQL not rendered in Rendered Templates view [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on PR #60739:
URL: https://github.com/apache/airflow/pull/60739#issuecomment-3889065300
Ptal @amoghrajesh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Bugfix: SQL not rendered in Rendered Templates view [airflow]
Posted by "nathadfield (via GitHub)" <gi...@apache.org>.
nathadfield commented on PR #60739:
URL: https://github.com/apache/airflow/pull/60739#issuecomment-3943826090
Thank you and thanks to @sarth-akvaish for the fix!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org