You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "eschutho (via GitHub)" <gi...@apache.org> on 2025/02/06 22:29:37 UTC
[PR] feat: add functionality for nested jinja rendering [superset]
eschutho opened a new pull request, #32175:
URL: https://github.com/apache/superset/pull/32175
<!---
Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/
Example:
fix(dashboard): load charts correctly
-->
### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
### TESTING INSTRUCTIONS
<!--- Required! What steps can be taken to manually verify the changes? -->
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: add functionality for nested jinja rendering [superset]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32175:
URL: https://github.com/apache/superset/pull/32175#issuecomment-2643085982
@dpgaspar Processing your ephemeral environment request [here](https://github.com/apache/superset/actions/runs/13201819347). Action: **up**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: add functionality for nested jinja rendering [superset]
Posted by "eschutho (via GitHub)" <gi...@apache.org>.
eschutho closed pull request #32175: fix: add functionality for nested jinja rendering
URL: https://github.com/apache/superset/pull/32175
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: add functionality for nested jinja rendering [superset]
Posted by "korbit-ai[bot] (via GitHub)" <gi...@apache.org>.
korbit-ai[bot] commented on code in PR #32175:
URL: https://github.com/apache/superset/pull/32175#discussion_r1945860279
##########
superset/jinja_context.py:
##########
@@ -596,12 +596,24 @@ def process_template(self, sql: str, **kwargs: Any) -> str:
>>> sql = "SELECT '{{ datetime(2017, 1, 1).isoformat() }}'"
>>> process_template(sql)
"SELECT '2017-01-01T00:00:00'"
+ This function will attempt to resolve nested Jinja expressions
+ up to 4 iterations.
"""
- template = self.env.from_string(sql)
+ # Initialize the rendered SQL with the original SQL string.
+ rendered_sql = sql
kwargs.update(self._context)
-
context = validate_template_context(self.engine, kwargs)
- return template.render(context)
+
+ # Render recursively up to 4 levels.
+ for _i in range(4):
+ template = self.env.from_string(rendered_sql)
+ new_rendered_sql = template.render(context)
+ # If rendering no longer changes the SQL, break early.
+ if new_rendered_sql == rendered_sql:
+ break
+ rendered_sql = new_rendered_sql
Review Comment:
### Redundant Jinja Template Compilation <sub></sub>
<details>
<summary>Tell me more</summary>
###### What is the issue?
The Jinja template compilation (from_string) is performed unnecessarily in each iteration, even when the template string hasn't changed.
###### Why this matters
Jinja template compilation is a relatively expensive operation that parses the template string and creates an internal representation. Recompiling the same template string multiple times adds unnecessary overhead.
###### Suggested change ∙ *Feature Preview*
Move the template compilation outside the loop when rendered_sql hasn't changed:
```python
rendered_sql = sql
kwargs.update(self._context)
context = validate_template_context(self.engine, kwargs)
# Render recursively up to 4 levels
for _i in range(4):
template = self.env.from_string(rendered_sql)
new_rendered_sql = template.render(context)
# If rendering no longer changes the SQL, break early
if new_rendered_sql == rendered_sql:
break
rendered_sql = new_rendered_sql
return rendered_sql
```
</details>
<sub>
[](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/cd11412c-bfa5-4080-a3fb-f2cb4f2c304a?suggestedFixEnabled=true)
💬 Chat with Korbit by mentioning @korbit-ai.
</sub>
<!--- korbi internal id:1f120a24-0b10-4492-84ca-9b57bfe21724 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: add functionality for nested jinja rendering [superset]
Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida commented on code in PR #32175:
URL: https://github.com/apache/superset/pull/32175#discussion_r1951578488
##########
superset/jinja_context.py:
##########
@@ -596,12 +596,24 @@ def process_template(self, sql: str, **kwargs: Any) -> str:
>>> sql = "SELECT '{{ datetime(2017, 1, 1).isoformat() }}'"
>>> process_template(sql)
"SELECT '2017-01-01T00:00:00'"
+ This function will attempt to resolve nested Jinja expressions
+ up to 4 iterations.
"""
- template = self.env.from_string(sql)
+ # Initialize the rendered SQL with the original SQL string.
+ rendered_sql = sql
kwargs.update(self._context)
-
context = validate_template_context(self.engine, kwargs)
- return template.render(context)
+
+ # Render recursively up to 4 levels.
+ for _i in range(4):
Review Comment:
I gave this more thought, and I think re-rendering the template might be problematic, since the template could end with template markers after being rendered, eg:
```python
Template("{{ foo }}").render(context={"foo": "{{"})
```
I think what we want to do is modify the `metric_macro` function to check if it has additional calls to `metric()` after being rendered, and if so, keep rendering the template.
https://github.com/apache/superset/blob/319a860f2375438448457892b3a841c83889fb1d/superset/jinja_context.py#L891
We can parse the template and check for the occurrence of `metric()` by analyzing the AST:
```python
from jinja2 import Environment, nodes
from jinja2.nodes import Call, Node
def has_metric_macro(template_string: str) -> bool:
env = Environment()
ast = env.parse(template_string)
def visit_node(node: Node) -> bool:
return (
isinstance(node, Call)
and isinstance(node.node, nodes.Name)
and node.node.name == "metric"
) or any(visit_node(child) for child in node.iter_child_nodes())
return visit_node(ast)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: add functionality for nested jinja rendering [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on PR #32175:
URL: https://github.com/apache/superset/pull/32175#issuecomment-2643681271
@eschutho Looking at the before screenshot, do you know why we are showing a network error instead of a parsing error in Explore?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: add functionality for nested jinja rendering [superset]
Posted by "eschutho (via GitHub)" <gi...@apache.org>.
eschutho commented on PR #32175:
URL: https://github.com/apache/superset/pull/32175#issuecomment-2643813162
> @eschutho Looking at the before screenshot, do you know why we are showing a network error instead of a parsing error in Explore?
Hi @michael-s-molina no, I don't, and definitely that seems like a new error that I haven't seen before. Didn't have a chance to dig into why that error is showing up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: add functionality for nested jinja rendering [superset]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32175:
URL: https://github.com/apache/superset/pull/32175#issuecomment-2641270516
@eschutho Processing your ephemeral environment request [here](https://github.com/apache/superset/actions/runs/13189120091). Action: **up**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: add functionality for nested jinja rendering [superset]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32175:
URL: https://github.com/apache/superset/pull/32175#issuecomment-2641285959
@eschutho Ephemeral environment spinning up at http://34.210.157.115:8080. Credentials are 'admin'/'admin'. Please allow several minutes for bootstrapping and startup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: add functionality for nested jinja rendering [superset]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32175:
URL: https://github.com/apache/superset/pull/32175#issuecomment-2643122274
@dpgaspar Ephemeral environment spinning up at http://52.13.5.59:8080. Credentials are 'admin'/'admin'. Please allow several minutes for bootstrapping and startup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: add functionality for nested jinja rendering [superset]
Posted by "korbit-ai[bot] (via GitHub)" <gi...@apache.org>.
korbit-ai[bot] commented on code in PR #32175:
URL: https://github.com/apache/superset/pull/32175#discussion_r1945566562
##########
superset/jinja_context.py:
##########
@@ -596,12 +596,24 @@ def process_template(self, sql: str, **kwargs: Any) -> str:
>>> sql = "SELECT '{{ datetime(2017, 1, 1).isoformat() }}'"
>>> process_template(sql)
"SELECT '2017-01-01T00:00:00'"
+ This function will attempt to resolve nested Jinja expressions
+ up to 4 iterations.
"""
- template = self.env.from_string(sql)
+ # Initialize the rendered SQL with the original SQL string.
+ rendered_sql = sql
kwargs.update(self._context)
-
context = validate_template_context(self.engine, kwargs)
- return template.render(context)
+
+ # Render recursively up to 4 levels.
+ for _i in range(4):
Review Comment:
### Fixed Iteration Limit May Be Insufficient <sub></sub>
<details>
<summary>Tell me more</summary>
###### What is the issue?
The hardcoded limit of 4 iterations may be insufficient for deeply nested expressions that require more resolution steps.
###### Why this matters
Users may have valid use cases requiring more than 4 levels of nesting, leading to incompletely resolved templates.
###### Suggested change ∙ *Feature Preview*
Make the iteration limit configurable and add validation:
```python
MAX_TEMPLATE_RESOLUTION_ITERATIONS = current_app.config.get('MAX_TEMPLATE_RESOLUTION_ITERATIONS', 4)
if MAX_TEMPLATE_RESOLUTION_ITERATIONS > 10:
raise SupersetTemplateException('Maximum template resolution iterations cannot exceed 10')
for _i in range(MAX_TEMPLATE_RESOLUTION_ITERATIONS):
template = self.env.from_string(rendered_sql)
rendered_sql = template.render(context)
```
</details>
<sub>
[](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/e4a174ae-8701-45ed-9392-03642eb4f56b?suggestedFixEnabled=true)
💬 Chat with Korbit by mentioning @korbit-ai.
</sub>
<!--- korbi internal id:4f70303e-dc4c-48f3-b94f-a5ecda10afc1 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: add functionality for nested jinja rendering [superset]
Posted by "eschutho (via GitHub)" <gi...@apache.org>.
eschutho commented on PR #32175:
URL: https://github.com/apache/superset/pull/32175#issuecomment-2655024279
Going with @betodealmeida's PR instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org