You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "pankajsoni22 (via GitHub)" <gi...@apache.org> on 2023/11/02 10:47:16 UTC
[I] Jinja execution is not happening when a dataset "sync columns" feature is used [superset]
pankajsoni22 opened a new issue, #25839:
URL: https://github.com/apache/superset/issues/25839
A clear and concise description of what the bug is.
#### How to reproduce the bug
1. Go to a dataset which implements time filter 'from_dttm'/'to_dttm' jinja values to use them in the dataset.
2. Click edit button on dataset
3. Click on third tab "COLUMNS" (Refer the screenshot below)
4. Click on "SYNC COLUMNS FROM SOURCE"
5. Query throws the error about syntax error because does not execute Jinja
### Expected results
It should sync the columns without any error
### Actual results
Generate SQL syntax error as Jinja is not native to SQL.
#### Screenshots
### Environment
Feature flags:
`ENABLE_TEMPLATE_PROCESSING=True`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Jinja execution is not happening when a dataset "sync columns" feature is used [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #25839:
URL: https://github.com/apache/superset/issues/25839#issuecomment-2860354551
@Vitor-Avila @sadpandajoe do you know if we resolved this one recently? I feel like we did improve some sync columns stuff, but I can't remember if this was part of it :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Jinja execution is not happening when a dataset "sync columns" feature is used [superset]
Posted by "Vitor-Avila (via GitHub)" <gi...@apache.org>.
Vitor-Avila commented on issue #25839:
URL: https://github.com/apache/superset/issues/25839#issuecomment-2860460371
I don't think we touched this part.
But generally curious, could this be solved with `if/else` statements that only add the Jinja block if they actually have a value? Like is the issue here that the query gets generated and those Jinja macros don't return any value in that context which produces a broken SQL? Or is the issue here that the Jinja templating syntax is not being properly rendered?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Jinja execution is not happening when a dataset "sync columns" feature is used [superset]
Posted by "biffhero (via GitHub)" <gi...@apache.org>.
biffhero commented on issue #25839:
URL: https://github.com/apache/superset/issues/25839#issuecomment-2591521614
I was able to perform a workaround. Create the query in SQL Lab, add the default parameters to the SQL Lab, verify it works. Save it, go to the dataset browser.
In the dataset browser, edit the dataset, unlock the query. Go to the query, cut out the Jinja parameters, sync the columns.
Replace the Jinja parameters, save the dataset. Go to the charts, the columns are available to use.
I agree with developers if they were to say "There's a workaround, this isn't a huge problem." I'd like to see the sync button submit the query without the Jinja parameters, read the column types, and then put the Jinja parameters back. Would that be possible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Jinja execution is not happening when a dataset "sync columns" feature is used [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #25839:
URL: https://github.com/apache/superset/issues/25839#issuecomment-2855529785
Can anyone confirm if this is in 4.1.2/5.0.0rc2/latest `master`? It's been silent for quite a while.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Jinja execution is not happening when a dataset "sync columns" feature is used [superset]
Posted by "sfirke (via GitHub)" <gi...@apache.org>.
sfirke commented on issue #25839:
URL: https://github.com/apache/superset/issues/25839#issuecomment-1791046586
Thanks for reporting. This behavior has existed for a while but I don't think is clearly documented like this. I use this workaround to sync columns: https://github.com/apache/superset/issues/21793#issuecomment-1384138580 Basically, delete the Jinja, sync columns, re-paste the Jinja again. But ideally having Jinja templating would not break the sync columns function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Jinja execution is not happening when a dataset "sync columns" feature is used [superset]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
dosubot[bot] commented on issue #25839:
URL: https://github.com/apache/superset/issues/25839#issuecomment-3643862476
<!-- Answer -->
The sync columns logic lives mainly in the backend in the `SqlaTable` class, specifically the `fetch_metadata` method in [`superset/connectors/sqla/models.py`](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/connectors/sqla/models.py#L818-L1867). This method fetches and merges column metadata from the database, but does not process or strip Jinja templating syntax before running the sync, which is why SQL errors occur when template variables are present in the dataset SQL.
On the frontend, the sync is triggered by the `syncMetadata` method in the `DatasourceEditor` component (`superset-frontend/src/components/Datasource/DatasourceEditor.jsx`), which calls `fetchSyncedColumns` and updates columns in state. Recent PRs have improved the UI and added options to skip syncing, but none address Jinja handling during sync columns—so the issue still exists in Superset 5.0+ as of the latest code and merged PRs [1](https://github.com/apache/superset/pull/30903), [2](https://github.com/apache/superset/pull/35650).
For anyone wanting to contribute a fix: the backend `fetch_metadata` method is the key place to add logic to temporarily strip or safely process Jinja syntax before syncing columns. The frontend logic is in `DatasourceEditor` and `DatasourceModal`, but the core issue is backend-side.
Let me know if you want a diagram of the sync flow or more details on specific files/methods!
<!-- Dosu Comment Footer -->
*To reply, just mention [@dosu](https://go.dosu.dev/dosubot).*
---
<sup>How did I do? [Good](https://app.dosu.dev/response-feedback/6cda8463-2a2d-4e31-a497-536f065f154a?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/6cda8463-2a2d-4e31-a497-536f065f154a?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/6cda8463-2a2d-4e31-a497-536f065f154a?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/6cda8463-2a2d-4e31-a497-536f065f154a?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/6cda8463-2a2d-4e31-a497-536f065f154a?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/6cda8463-2a2d-4e31-a497-536f065f154a?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/6cda8463-2a2d-4e31-a497-536f065f154a?feedback_type=other)</sup>
[](https://app.dosu.dev/a28d3c7e-a9d3-459e-9fb6-3a6f9ff4f357/ask?utm_source=github) [](https://cursor.com/link/prompt?text=The%20sync%20columns%20logic%20lives%20mainly%20in%20the%20backend%20in%20the%20%60SqlaTable%60%20class%2C%20specifically%20the%20%60fetch_metadata%60%20method%20in%20%5B%60superset/connectors/sqla/models.py%60%5D%28https%3A//github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/connectors/sqla/models.py%23L818-L1867%29.%20This%20method%20fetches%20and%20merges%20column%20metadata%20from%20the%20database%2C%20but%20does%20not%20process%20or%20strip%20Jinja%20templating%20syntax%20before%20running%20the%20sync%2C%20which%20is%20why%20SQL%20errors%20occur%20when%20template%20variables%20are%20present%20in%20the%20dataset%20SQL.%0A%0AOn%20the%20frontend%2C%20the%20sync%20is%20triggered%20by%20the%20%60syncMetadata%60%20meth
od%20in%20the%20%60DatasourceEditor%60%20component%20%28%60superset-frontend/src/components/Datasource/DatasourceEditor.jsx%60%29%2C%20which%20calls%20%60fetchSyncedColumns%60%20and%20updates%20columns%20in%20state.%20Recent%20PRs%20have%20improved%20the%20UI%20and%20added%20options%20to%20skip%20syncing%2C%20but%20none%20address%20Jinja%20handling%20during%20sync%20columns%E2%80%94so%20the%20issue%20still%20exists%20in%20Superset%205.0%2B%20as%20of%20the%20latest%20code%20and%20merged%20PRs%20%5B1%5D%28https%3A//github.com/apache/superset/pull/30903%29%2C%20%5B2%5D%28https%3A//github.com/apache/superset/pull/35650%29.%0A%0AFor%20anyone%20wanting%20to%20contribute%20a%20fix%3A%20the%20backend%20%60fetch_metadata%60%20method%20is%20the%20key%20place%20to%20add%20logic%20to%20temporarily%20strip%20or%20safely%20process%20Jinja%20syntax%20before%20syncing%20columns.%20The%20frontend%20logic%20is%20in%20%60DatasourceEditor%60%20and%20%60DatasourceModal%60%2C%20but%20the%20core%20issue%2
0is%20backend-side.%0A%0ALet%20me%20know%20if%20you%20want%20a%20diagram%20of%20the%20sync%20flow%20or%20more%20details%20on%20specific%20files/methods%21) [](https://go.dosu.dev/discord-bot) [](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/superset/issues/25839)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Jinja execution is not happening when a dataset "sync columns" feature is used [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #25839:
URL: https://github.com/apache/superset/issues/25839#issuecomment-3643852211
This issue has been open for over 2 years with confirmed reproductions. The workaround (remove Jinja → sync columns → re-add Jinja) works but is cumbersome.
**Status check:** Can anyone confirm if this still occurs on **Superset 5.0+**? If so, this would be a good candidate for contribution - the fix would likely involve temporarily stripping Jinja syntax before running the column sync query.
@dosu - can you help identify where the sync columns logic lives in the codebase for anyone wanting to pick this up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Jinja execution is not happening when a dataset "sync columns" feature is used [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #25839:
URL: https://github.com/apache/superset/issues/25839#issuecomment-2040307836
It's been about 5 months since this thread was last touched. Can anyone confirm if the issue still exists in 3.1.2 or 4.0? Otherwise this is at risk of being closed as stale, in assumption that the workaround is good enough for now :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Jinja execution is not happening when a dataset "sync columns" feature is used [superset]
Posted by "aolwas (via GitHub)" <gi...@apache.org>.
aolwas commented on issue #25839:
URL: https://github.com/apache/superset/issues/25839#issuecomment-2041008990
The issue still exists at least in 3.1.1.
I can't tell if a fix is available in last releases just looking at the changelogs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Jinja execution is not happening when a dataset "sync columns" feature is used [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas closed issue #25839: Jinja execution is not happening when a dataset "sync columns" feature is used
URL: https://github.com/apache/superset/issues/25839
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org