You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "pomegranited (via GitHub)" <gi...@apache.org> on 2025/03/26 05:31:49 UTC
[I] [SIP] Translating Superset asset data [superset]
pomegranited opened a new issue, #32854:
URL: https://github.com/apache/superset/issues/32854
## [SIP] Proposal for Translating Superset asset data
### Motivation
Superset provides translation support for built-in components in the UI. However, the Open edX project also needs the user-provided terms used in the assets themselves to be translatable, e.g. dashboard and chart title, axes labels, and metric labels.
We also need these asset translations to be easily maintained between upgrades of Superset, and to be re-deployable when translations are updated. An automatic way to export translated asset fields and import translations would also help integrate Superset into Open edX's ecosystem.
Superset's requirements for this feature include (refs [1](https://github.com/apache/superset/issues/32139#issuecomment-2675387707), [2](https://github.com/apache/superset/issues/32139#issuecomment-2670053442)):
* Zero performance hit for English `en` (or when running with a single system default language)
The system default language strings should live in the field itself, so no extra lookups are required.
* No data translations will be added to Superset's officially supported translations -- just like asset data, these translations must be managed by Superset users.
* Translators can update asset translations in the Superset UI and see their updates in real time.
Desirable features include:
* Searchability of the appropriate translated text when user has selected a non-`en` language ([ref](https://github.com/apache/superset/issues/32139#issuecomment-2675846271)).
Superset, by default, is configured to use English (`en`) as the single default language via the [`BABEL_DEFAULT_LOCALE`](https://github.com/apache/superset/blob/45ea11c1b65887755f30e5945ea280abc0847929/superset/config.py#L374) setting. Some translations for built-in UI components are provided for 16 additional languages which can be enabled via the [`LANGUAGES`](https://github.com/apache/superset/blob/45ea11c1b65887755f30e5945ea280abc0847929/superset/config.py#L378-L396) setting, but [these languages are disabled by default](https://github.com/apache/superset/blob/45ea11c1b65887755f30e5945ea280abc0847929/superset/config.py#L397-L399) for performance reasons, and due to incomplete available translations.
Open edX aims to implement this feature to increase uptake of the use of Superset in our ecosystem. The more internationalised sites that run Superset, the more people will be motivated to fill in Superset's missing UI translations for more supported languages.
### Proposed Change
We propose implementing this feature in two phases.
This feature will be disabled by default, and can be enabled by an app-wide feature flag.
#### Phase 1: Functionality
Add the base functionality for rendering configured translatable model fields using a new jinja filter called `i18n`.
No changes to the UI or data will be visible to a user who edits a translatable field -- the user continues to provide the full `BABEL_DEFAULT_LOCALE` default text for the field.
When this feature is enabled and the user views an asset, they will see their selected language's translation, if found, or the default text, if not.
When rendering translatable fields when this feature is enabled:
* If there's only one language configured, or the user is using the app's default language, simply return the field's default text.
* Else:
* Show `translated_text` found in the `i18n_translations` table for the given asset UUID + default text + user's current language.
* Fallback to the default text.
#### Phase 2: UX
Add buttons to the Superset UI to enable editors to provide translations of asset text in the Superset UI.
Exact UX TBD, but see https://github.com/apache/superset/issues/13442 for one nice way to do this.
When editing a templated field, the user should be able to see:
* The full `default_text` value to be translated
* All configured [`LANGUAGES`](https://github.com/apache/superset/blob/45ea11c1b65887755f30e5945ea280abc0847929/superset/config.py#L378-L396) to be translated
This phase requires making a lot of UI changes, so will likely be broken into smaller contributions.
It also would benefit from the input of a UI/UX designer, which we can provide.
### New or Changed Public Interfaces
* Visualization types -- unchanged
* Form data for saved dashboards and charts -- likely unchanged, but see "Open Question 4" below.
#### Phase 1: Functionality
Configuration:
1. A new configuration feature flag would be added to enable/disable this feature, called `ASSET_TRANSLATIONS`.
Data passed between backend and frontend:
1. Add new jinja filter called `i18n` which uses the `i18n_translations` table described below to render translated text in a request.
2. Add a model mixin to add the concept of "templated fields"
E.g the Dashboard model would have the following class variable:
```python
templated_fields = [
'dashboard_title',
'description',
'json_metadata',
'position_json',
]
```
3. When rendering model fields to the "view" API:
* If the app-wide setting is enabled: wrap templated field values in the `i18n` jinja filter, and render values.
* Else: pass field values straight through.
Command line tools and arguments:
1. New command-line tools will be added to extract translatable asset fields and import translated text into the `i18n_translations` table, using the `.mo` translation file format.
#### Phase 2: UX
A new "translation UI component" would take these properties as input:
* `asset_uuid`
* `model_name`
* `field_name`
* `default_text`
REST endpoints:
1. A "translation API" would be added to the backend to support these components for viewing and editing translated text for a given asset model field with the configured `BABEL_DEFAULT_LOCALE` and `LANGUAGES` settings.
### New dependencies
No new `npm`/`PyPI` packages required.
### Migration Plan and Compatibility
A migration will be added to create a new table called `i18n_translations`.
Fields will be:
* `asset_uuid`: unique identifier for the asset being translated.
* `default_text`: user-provided text for the configured `BABEL_DEFAULT_LOCALE`
* `language_code`: language/locale code for the translated text
* `translated_text`: `default_text` translation for the `language_code`
* `model_name`: name of the table containing the translatable field.
* `field_name`: name of the field being translated.
Indexes:
* `asset_uuid` + `default_text` + `language_code` (unique): used to locate any `translated_text` fields when rendering a given asset.
May also add `model_name` and/or `field_name` to this index, if needed.
Notes:
* Dashboards and charts that are saved or bookmarked will still work after the change, or when this change is enabled/disabled.
### Open Questions:
1. May need to batch queries or use a cache for performance.
2. Some model fields are edited within a modal popup – so we'd want to avoid a modal-within-modal UI.
3. Is there a difference in how echart editors are rendered vs old-style charts? i.e. Do they both use the data API?
4. Unsure how to provide translations for complex fields like dashboard [`json_metadata`](https://github.com/apache/superset/blob/45ea11c1b65887755f30e5945ea280abc0847929/superset/models/dashboard.py#L141), [`position_json`](https://github.com/apache/superset/blob/45ea11c1b65887755f30e5945ea280abc0847929/superset/models/dashboard.py#L136) and slice [`params`](https://github.com/apache/superset/blob/45ea11c1b65887755f30e5945ea280abc0847929/superset/models/slice.py#L80), e.g. `metadata.native_filter_configuration.name`, `position.*.meta.text`, `params.x_axis_label`.
May need to store templated field values in these complex fields, and mark the whole field (e.g `json_metadata`) as a `templated_field`.
5. Unsure how to provide asset search/filtering for translated text -- can we use [base_filters](https://flask-appbuilder.readthedocs.io/en/latest/advanced.html#base-filtering) somehow?
### Rejected Alternatives
* [SIP-60](https://github.com/apache/superset/issues/13442) -- The frontend UI proposed in SIP-60 could be used here. But we propose an alternative backend implementation which uses a single `i18n_translation` table to store all translations instead of introducing individual `<field_name>_i18n` field for each translatable field.
* [SIP-153](https://github.com/apache/superset/issues/32139) -- rejected because its approach was ["build time" focused and not as dynamic as desired](https://github.com/apache/superset/issues/32139#issuecomment-2670053442) and because it abuses the [frontend language packs shipped from the backend](https://github.com/apache/superset/issues/32139#issuecomment-2670069005), which is not ideal. SIP-153 also didn't support returning translated fields in the data APIs, while this solution does.
* Decided not to support the more complex jina filter format [proposed here](https://github.com/apache/superset/issues/32139#issuecomment-2675387707) e.g `"CompanyX {{ "Revenue Dashboard" | i18n("rev_dash") }}"` because:
1. Many languages would need to change the order of the translatable phrases in order to be sensible, e.g. "Panel de ingresos de la CompanyX", and so translating the full phrase is better.
2. Simplifies the editing user's experience by not requiring them to use jinja template syntax
3. Allows this feature to be enabled or disabled without having to change data. E.g. if the actual stored `dashboard_title = "CompanyX {{ "Revenue Dashboard" | i18n("rev_dash") }}`, this field becomes unintelligible if the feature is subsequently disabled.
4. We have enough information (asset UUID, model name, field name) to uniquely identify the translated field text if we treat it as an atomic unit, rather than allowing its value to be broken into some unknown number of parts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset data [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-2791220088
NOTE: Directionally, I think the folks that were part of the conversation want to build minimal hooks for people to build-upon, and leave functionality to the extension implementation - note that extensions can still be open sourced and shared within the community, but directionally trying to contain the core project's surface. Question becomes more about offering the right interfaces for people building extensions.
Out of all of this I like the idea of `templated_fields` in core Superset models to allow for dynamic / extensible model properties. Some of what it enables - beyond i18n:
- Rich/dynamic descriptions with AI subsystem where say a chart description could be `{{ llm_interface_describe_chart(chart_object) }}`
- reference object properties in titles: `My dashboard (as of {{ obj.last_modified }})`
- i18n composition: `{{ ai18n("Dashboard") }} - {{ ai18n("by month") }}`, where smaller - more atomic strings can be instrumented for translations
- showing chart defining filters in titles: `Population by country for {{ main_time_filter.humanize() }}`
- dynamically reference metric or dimension names in chart titles `{{ main_metric.label }} by {{ main_dimension.label }}`
- surfacing things like when the dashboard was rendered in its title `Incident report as of {{ now().isoformat() }}` - say for alerts and reports
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset metadata [superset]
Posted by "pomegranited (via GitHub)" <gi...@apache.org>.
pomegranited closed issue #32854: [SIP-161] Translating Superset asset metadata
URL: https://github.com/apache/superset/issues/32854
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset metadata [superset]
Posted by "khashashin (via GitHub)" <gi...@apache.org>.
khashashin commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-3861722544
@rusackas would you consider reopening the issue in order to pursue this feature further?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset metadata [superset]
Posted by "khashashin (via GitHub)" <gi...@apache.org>.
khashashin commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-3608979547
I’d recommend keeping this open rather than marking it abandoned. Metadata-level i18n is a recurring need for real deployments, especially as Superset grows into more global and multi-tenant environments. The earlier idea of a lightweight TRANSLATION_HOOK seems like exactly the kind of minimal, pluggable interface that lets core remain simple while still supporting multilingual and multi-tenant setups.
A hook like that imposes almost no long-term maintenance cost on the project, stays completely optional, and solves a widespread problem that many production deployments encounter as soon as they start supporting multiple audiences or tenants. Given how common this requirement is for platforms that aim to scale internationally, it feels premature to mark the discussion as abandoned.
If there’s still interest from anyone in exploring this direction, a small RFC capturing the hook signature and behavior might be enough to continue moving forward. I’m happy to help review or provide feedback if that would be useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset metadata [superset]
Posted by "mattjmitchell (via GitHub)" <gi...@apache.org>.
mattjmitchell commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-3676096442
I am definitely interested in this feature - I've landed here because I'm specifically trying to solve the same use-case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset data [superset]
Posted by "pomegranited (via GitHub)" <gi...@apache.org>.
pomegranited commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-2804163862
@rusackas
> Do you think you might be able to join the Superset Town Hall on Friday at 9am PST? Maybe it'll be easier to go through some of this more synchronously, but we can keep chatting here (and on the mailing list) as warranted :)
Sorry, I'm not able to make that the Town Halls due to my timezone UTC+9:30.
But thank you for continuing to discuss this here!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP] Translating Superset asset data [superset]
Posted by "pomegranited (via GitHub)" <gi...@apache.org>.
pomegranited commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-2759740921
Great suggestion @nytai !
I don't think I can provide that functionality as part of this SIP, but having the UI in place at least opens the possibility.
The command line export/import feature also provides an opportunity for machine translations, e.g using Transifex (which is [free for open source](https://www.transifex.com/open-source/)).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset metadata [superset]
Posted by "pomegranited (via GitHub)" <gi...@apache.org>.
pomegranited commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-2846537280
@rusackas I think much of the functionality here could be done as a plugin/extension, to keep the logic separate from Superset's application logic.
But I see there's 2 types of application plugin formats: [Flask extension](https://flask.palletsprojects.com/en/stable/extensions/) or [Flask AppBuilder Add-On](https://flask-appbuilder.readthedocs.io/en/latest/addons.html).
Are they different? Do you have a preference for which I use?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset data [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-2791104480
Do you think you might be able to join the Superset Town Hall on Friday at 9am PST? Maybe it'll be easier to go through some of this more synchronously, but we can keep chatting here (and on the mailing list) as warranted :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset data [superset]
Posted by "pomegranited (via GitHub)" <gi...@apache.org>.
pomegranited commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-2772057970
@jpchev
> as far as I understand the feature is to let dashboard creators provide a translation in multiple languages (localisation) for their content (chart title, descriptions....), right?
Yep!
> I don't need this for my projets, but I think it's very interesting and adding value to the Superset product.
Thank you! I thought I'd mention it just in case it was useful to you.
> I don't know what is open edx, though, is it a project you are working on?
Oops! I forgot to add a link and details, have updated the description here. And yep, Open edX is the reason I'm working on Superset, since we use it for our learner analytics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset metadata [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-5458675508
Just put this up for a vote. Thanks @alex-poor for shepherding this, and thanks to everyone here for your patience.
Here's the summary I posted to the dev@ list to recap on the Vote:
**What new things can be translated?**
- Chart names and dashboard titles, automatically — no author action needed.
- Home-page "Recents" item titles.
- Anything an author explicitly wraps in a new `{{ i18n('...') }}` Jinja macro inside a templated field (axis labels, native filter names, etc.)
- This is opt-in per field, not automatic.
- Explicitly not covered: the actual chart data/values and localized number/date formatting, which are the same exclusions the original SIP had.
**How are translations maintained and loaded?**
- Superset core stores none of it, that's the design shift from the original proposal.
- There's one hook:
- TRANSLATION_HOOK(text, locale, **context) -> str | None
- You implement it in config, and back it with whatever you want — a static dict, an external MT API, gettext `.po` catalogs, or a database table, whatever
- A `TRANSLATION_BATCH_HOOK` variant exists for anything where each lookup is a query or network call. Superset prefetches through it wherever it already holds a full collection (a dashboard's charts, the Recents list). That means a 30-chart dashboard costs 1 hook call, not 30, and results are memoized. The one place that still resolves per-row is the list views, since FAB serializes each row independently. The docs tell you to cache your hook if that matters.
- There's no export/import built into core; that's on whatever you build behind the hook.
**Effect on our `.pot`/`.po` effort?**
- None on the existing pipeline — UI chrome strings are unaffected, nothing new gets added to Superset's own translatable-string extraction.
- One of the four reference hooks in the docs *does* show how a deployment could reuse the gettext toolchain (Poedit/Transifex/Weblate) for asset content. But that’s in a separate catalog domain (e.g. `assets`) so it can't collide with or pollute our existing UI catalogs. Opt-in per deployment, zero impact on Superset’s core.
**Breaking? Nope… fully additive.**
- Feature flag + a multi-language `LANGUAGES` config are both required to activate anything, so virtually every existing single-language install is untouched.
- New optional config hooks (no-op if undefined), new dump-only REST fields (`localized_name`/`localized_title`, nothing removed or renamed)
- One new Jinja macro
- No DB migration.
- Editing a name always shows/saves the canonical value, the edit workflow doesn't change.
**Summary of what you'd be voting on:**
- A `TRANSLATION_HOOK(text, locale, **context) -> str | None` config hook (plus an optional batch variant for query/API-backed stores), fully pluggable — static map, external MT service, `.po` catalogs, a DB table, whatever a deployment wants.
- A `{{ i18n('...') }}` Jinja macro for opt-in translation of templated fields.
- Gated behind `ENABLE_I18N_ASSET_TRANSLATIONS` + a multi-language `LANGUAGES` config, so single-language deployments (the default) pay zero cost.
- No new core dependencies, no DB migration, no translations stored in Superset itself. Fully additive REST fields, editing always operates on the canonical name.
- Automatically covers chart names and dashboard titles; anything else (axis labels, native filter names, etc.) is opt-in via the macro. Search/filtering on translated text is not addressed. Both are known, named gaps relative to the original SIP's ambitions, not oversights.
**Worth flagging from prior discussion / alternatives:**
- The original motivation explicitly called out *axis and metric labels* as things to translate. This PR only auto-covers `slice_name`/`dashboard_title`. Everything else needs manual `{{ i18n() }}` wrapping per field, per org, so it’s a narrower scope
- "Search/filter by translated text" was an open question in the original SIP and remains unaddressed — list search still matches canonical text only.
- The SIP's "markup vs. auto-wrap" contention is resolved as a hybrid option (auto for the two title fields, manual macro for everything else)
- It was doubted that batch retrieval was feasible *through Jinja* but this PR sidesteps that by prefetching outside the macro (dashboard-charts endpoint, Recents), while the macro itself still resolves one string per call site, consistent with rather than disproving the doubt.
The PR itself has already had thorough review already — worth an additional pass if you want the engineering details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP] Translating Superset asset data [superset]
Posted by "nytai (via GitHub)" <gi...@apache.org>.
nytai commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-2759577306
Interesting proposal, might be worth considering during the design phase for the UX component to edit the various languages, to provide a hook to call out to a provider like google translate. That would make it very easy to populate these translations while still having a human reviewing the quality before saving. Such a hooks should be implemented in a vendor agnostic way so the superset admin can choose their preferred provider. It could be as simple as providing a callable in the config file to look up the translation, similar to the `DB_CONNECTION_MUTATOR` or `SQL_QUERY_MUTATOR` configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset data [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-2776264375
Loving this! Lots of folks have been asking for this, and it's very much along the lines of what @mistercrunch and I were discussing a couple days ago, and we've talked about with others in the past. TL;DR: I'd love to see this happen!
@pomegranited
* > May need to batch queries or use a cache for performance.
This seems essential to me... otherwise I think we'll see some weird performance hits, both on the site load time side, and on the metadata db's stress level.
* > Some model fields are edited within a modal popup – so we'd want to avoid a modal-within-modal UI.
@kasiazjc might have good ideas. I believe the AntD Popover can support form elements, so that may work in all cases. The design trick might be to find some sort of mechanism to open this popover on hover or right-click so we don't drive people crazy with popovers appearing everywhere.
* > Is there a difference in how echart editors are rendered vs old-style charts? i.e. Do they both use the data API?
The legacy charts do use a different endpoint (which has been deprecated _forever_, but is a long tail of work)
* > Unsure how to provide translations for complex fields like dashboard json_metadata...
I'm not exactly sure either, but it *seems* plausible if we can find a way to templatize some of the JSON using HandleBars/Jinja.
@nytai
* > provide a hook to call out to a provider like google translate... in a vendor-agnostic way
1000% agreed. There may actually be many options for this, whether it's an Google Translate, a managed translation provider like Transifex/CrowdIn, or even LLMs. There have been MANY people asking where/how to integrate LLMs, and this seems like one case where we _might_ be able to add LangChain as a dependency, and let people configure their model. Any of these approaches would require the UI to have some sort of "enter or approve" option to place the appropriate translation into the DB table.
### Additional thoughts
I'm not sure where this might lead in the future, but I think there are some intriguing possibilities. Once this is is all proven out as laid out in the proposal, it seems like there would be a LOT of room for it to grow across the product. Namely, nobody is *really* in love with the current translation workflow. It's a standard, but it feels... antiquated, and has proven a bit fragile (i.e. if someone updates the frontend code with a new/modified string, it breaks existing translations). I could foresee a world in which ALL translated strings live in this table. The challenge would be to make sure that the UI could accumulate the required strings and fetch them in a batch... with some sort of graceful loading so it doesn't look janky. Then _all_ the UI strings could benefit from the translation (popover?) UI. Extending on that, when someone fills or corrects a "built in" string (as opposed to a dynamic one) it could be flagged as such in the table. Then it doesn't seem like a st
retch to have a script that would allow people to export a migration that could be used to contribute these translations (as a PR) for everyone to benefit from.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset metadata [superset]
Posted by "nieuwefn (via GitHub)" <gi...@apache.org>.
nieuwefn commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-3035055791
Any update on this?
Is there a draft PR anywhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset metadata [superset]
Posted by "alex-poor (via GitHub)" <gi...@apache.org>.
alex-poor commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-4600200170
Hi all — I know this was closed, but given the continued interest (and @khashashin's note about reopening) and my own emergent need for this, I've tried to pick it up and opened a draft PR: apache/superset#40679. If there's still appetite then I hope this might be a good reference point to move it forward.
I read above comments carefully, and the draft PR follows the direction this thread converged on (rather than the original table-based design):
- A configurable `TRANSLATION_HOOK` `(default_text, locale, **context) -> str | None` is the only integration point. **Superset core does not store translations** — the hook decides where they live (static map, external MT service, a DB table, gettext `.po` catalogs, …), keeping core minimal as @mistercrunch and @rusackas suggested.
- A `{{ i18n('...') }}` Jinja macro for templated fields.
- Gated by an `ENABLE_I18N_ASSET_TRANSLATIONS` feature flag and a multi-language `LANGUAGES` config, so single-language deployments pay nothing.
- The canonical stored name is always preserved — edit flows operate on the canonical value, so a translation can never be persisted over the real name. Missing/erroring hook falls back to the original text.
Localized surfaces: chart/dashboard lists & cards, dashboard header, chart panel titles, "Recents" cards, and `{{ i18n() }}`-wrapped fields. Storage and authoring UI are deliberately out of core; the PR includes a self-contained DB-backed reference under `examples/` to show end-to-end authoring.
**Scope note — what the draft PR does not cover yet**: The original proposal envisioned translating all chart text including axis labels and metric labels. This PR deliberately starts narrower: it auto-translates the names/titles that appear across the UI — slice_name, dashboard_title, and recent-activity item titles — plus anything an author explicitly wraps in the new {{ i18n('...') }} Jinja macro. It does not yet automatically localize axis titles, metric labels, column verbose_names, or native-filter names, since those live inside chart params/form_data and dataset metadata and need a per-field strategy. The resolver and hook are designed to extend to those fields; I'd propose landing the names/titles read path first and layering the deeper form_data/label coverage in a follow-up, so each step stays reviewable.
I'm happy to take this through the proper SIP process — should I (re)start a `[DISCUSS][SIP-161]` thread on `dev@`, or is reviewing the draft PR here the better next step? Thanks to @pomegranited and @bmtcril for the original proposal and interesting design discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset data [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-2790263715
Clearly, this is ready to bring into a [DISCUSS] thread on the `dev@` mailing list. Sounds like ideas are starting to align here, so perhaps some open questions are where the burden lives for translating titles and other dynamic/user strings - should it be in the app UI, or should it expose an API/DB to manage these translations extnerally?
Anyway, let me know if you need help opening the thread. I'm easy to find on Slack :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset data [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-2790649473
From a deep conversation with PMCs today - there seemed to be traction for a simpler solution around a more generic "translation hook", where administrators would be free to implement a generic function to hook up with an external translation service. The high level idea is that you would have some sort of `TRANSLATION_HOOK(text_to_be_translated, context_object)` function defined in something like `superset_config.py` and externalize the translation process through that. That hook would assume the external service is in charge of accumulating strings to be translated, memoization, and serving the text -> do whatever makes sense to you in there.
There were deep discussions about whether we're talking "all or nothing" approach, asking whether - in a given superset instance - all metadata or none should be flagged for translation. My take on this is that I think there needs to be a way to flag which strings/objects need or don't need to be translated - whether it's through jinja intrumentation, or property checkboxes "flag this dashboard for translation". Also some talks/questions about whether the administrator decides, the user decides or the external service decides what need to be translated.
On my side, and stepping back a little - I'm still seeing some benefits to the general idea of "framework support for jinja templated fields in Superset models", where we define a set of dynamic fields (through things like `Datashboard.template_fields = ['dahboard_title', 'description']`), and allow people in their instance to overload the jinja context with functions like` i18n` or `dashboardMetadataAccessor` or whatever make sense. This would open and widely solve "dynamic properties" across the board.
One more thing. Nothing here yet solves translating external data coming from the database, meaning there's still a gap in the solutioning (assuming the goal is 100% i18n). Sure you can translate the title of the dashboard, the name/label of the dimension and metrics, but what about the content returned by the database? A simple example would be:
- dashboard name "Demographics by country" -> "Demographiques par pays"
- dimension name "Country" -> "Pays"
- metric name "Birth rate" -> "Taux de naissance"
- but what about DATA!?!?!?? "United States" -> "Etats-Unis" - are we disregarding something important here? As in we go through all this trouble and it's impossible to achieve 100% i18n... Do we need a `DATAFRAME_I18N_HOOK`? Or does it even make sense to send result sets cells through `TRANSLATION_HOOK`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset data [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-2780156812
There's a lot to parse here, but will dump more thoughts:
### Clarifying the use of `string_id`s and/or use strings as keys in `i18n_translations`.
Say if I want to set a string for translation, I'd put something like `{{ i18n('Amazing Dashboard') }}`, but can consider also supporting `{{ i18n(string_id='AMZ_DASH', main_locale='Amazing Dashboard') }}`. There are tradeoffs here but either work. I would suggest against `{{ i18n(string_id='AMZ_DASH') }} where main locale has to be looked up, but hey, could be manageable too. Allowing **only** `{{ i18n('Amazing Dashboard') }}` is probably my preferred approach, though if you change something minor in the string, like capitalization, it orphans all the related translation, which is probably right (capitalization should change there too, but might have to copy/paste/edit for long strings).
### About caching
Worried about lots of IO with `i18n_translations`, I'd say it's probably reasonable for the backend to manage an in-memory data structure that it looks up first prior to fetching/caching from `i18n_translations`. Maybe that's a config flag `ENABLE_I18N_IN_MEMORY_CACHE` for those who prefer IO over memory footprint. Simple tuple used as keys as in `i18n_cache[(str_id, locale)]`, if it's there it's free to lookup, if not it fetches from `i18n_translations`. Memory footprint is probably reasonable here, a simple factor of (avg_string_size * number_of_strings * number_of_locales). Note that each web worker will have this incremental memory footprint, provision your instances accordingly.
### About batch-retrieval
I think that's hard with jinja, not sure if it's even possible. With in-mem caching it's not as needed as each lookup is super cheap.
### About auto-populating translation through a service
In the age of AI, I think no one should manually translate stuff anymore. The jinja macro, as it encounters new strings, will auto-populate rows in `i18n_translations` and flag them as "to-be-translated". It seems writing a simple job, or generating async jobs to call a service using celery-beat (a cron that can run every N minutes that looks for new string, batches a call to GPT or google translate). No UI required then. Even if we do want/need a UI for translation, i'd suggest building it externally to Superset maybe (similar to the poedit approach), so we'd only have to expose a CRUD REST API (if even) as you could slap a simple UI on top (Claude can probably build one quickly, or even plug a simple no-code solution like ReTool or AirTable on top of that table/model). It punts this UI complexity outside of Superset.
### About garbage collection
Lots of strings might get orphaned, and can be good to trim the table. That would require maintaining some sort of "last_used_dttm" thing, but that's more IO to manage... Maybe it's yet another config flag, where periodically could turn it on, say for a month, and delete based on usage... Maybe I'm overthinking it. Or maybe it's lazy, like if we read the string and notice that it's more than say a week since the last recorded read, only then you run an UPDATE for that `last_used_dttm`, minimizing write operations on read.
### About Jinja overhead
Curious how much the `templated_fields` approach, even in `en`-only envs add overheads. You have to create the jinja template object and `.render()` it, which even if it's just a short "oh I'm in en, don't have to do anything here...", it may add significant overhead. I'd suggest either profiling this, or simply not bothering with doing the template logic if/when we're in a non-i18n environment.
-------
All in all, a few new ideas here and things to clarify. Main thing I'd push hard on is keeping the "translation UI" out of Superset, maybe even the translation service logic as there are a bunch of non-open-source services we'd have to rely on... Maybe it's a plugin or something that lives outside of Superset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset metadata [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-2898619847
The issue here is that the main maintainers haven't aligned on how exactly to do this, so it hasn't become a priority yet to come up with the perfect design here. I'm guessing you don't want to wait an extended time for resolution here, so another approach might be to open a (Draft) PR addressing this however works for your org, and then you can use that on your fork for the moment, and we can decide if we want to merge/maintain it as is... there's just a risk of it becoming something we want to redesign later, which creates some technical debt risk. We don't want to make you wait and lose the contribution, so perhaps that's the better approach for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset metadata [superset]
Posted by "thib-d (via GitHub)" <gi...@apache.org>.
thib-d commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-5437154718
Looking forward of the merge @alex-poor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset data [superset]
Posted by "pomegranited (via GitHub)" <gi...@apache.org>.
pomegranited commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-2777326668
Thank you for your support @rusackas !
> > * May need to batch queries or use a cache for performance.
>
> This seems essential to me... otherwise I think we'll see some weird performance hits, both on the site load time side, and on the metadata db's stress level.
The extra fields I've added to the `i18n_translations` table support this (`asset_uuid`, `model_name`).
I also think these fields might provide foreign key-like relationships that we can use for searching/filtering on translated terms.
>> * Some model fields are edited within a modal popup – so we'd want to avoid a modal-within-modal UI.
>
> @kasiazjc might have good ideas. I believe the AntD Popover can support form elements, so that may work in all cases. The design trick might be to find some sort of mechanism to open this popover on hover or right-click so we don't drive people crazy with popovers appearing everywhere.
Those popovers look lovely: https://ant.design/components/popover
Is there anyone else in the community whom we should draw on for UX advice? We've got UX experts here at my company, OpenCraft, but they're not Superset experts (yet 😄 ).
>> * Is there a difference in how echart editors are rendered vs old-style charts? i.e. Do they both use the data API?
>
> The legacy charts do use a different endpoint (which has been deprecated forever, but is a long tail of work)
Noted.. Is there a ticket where I can follow the deprecation process and procedure for migrating?
Should I avoid adding this feature to the deprecated endpoint? It would reduce the UX work as well to only support translating e-charts. [Open edX currently uses some legacy charts](https://github.com/openedx/openedx-aspects/issues/314), but we will migrate off of them, especially if this feature is e-chart only.
>> * provide a hook to call out to a provider like google translate... in a vendor-agnostic way
>
> 1000% agreed. There may actually be many options for this, whether it's an Google Translate, a managed translation provider like Transifex/CrowdIn, or even LLMs. There have been MANY people asking where/how to integrate LLMs, and this seems like one case where we might be able to add LangChain as a dependency, and let people configure their model. Any of these approaches would require the UI to have some sort of "enter or approve" option to place the appropriate translation into the DB table.
I've seen that [Superset supports visualization plugins](https://superset.apache.org/docs/contributing/howtos/#creating-visualization-plugins), but that seems like a different plugin architecture than what would be required to plug in a machine translation provider? Can you point me to any code examples of a UI plugin that isn't a visualization? Depending on how hard this is, I'd be happy to include hooks for future machine translators as part of Phase 2.
> * I'm not sure where this might lead in the future, but I think there are some intriguing possibilities. Once this is is all proven out as laid out in the proposal, it seems like there would be a LOT of room for it to grow across the product. Namely, nobody is really in love with the current translation workflow. ... I could foresee a world in which ALL translated strings live in this table.
Sure, I could see that happening.. we could support it with custom babel extractors like I proposed using in https://github.com/apache/superset/issues/32139, to keep the separation between "app" translations and "asset" translations.
> It's a standard, but it feels... antiquated, and has proven a bit fragile (i.e. if someone updates the frontend code with a new/modified string, it breaks existing translations).
That's always a challenge when changing the source string.. it can be hard to know when it's big enough change to require re-translation, and indicating that to the translation system can be tricky.
Note that we'll have the same issue here -- I've proposed using the English source string as part of the lookup key. I could use the `asset_uuid` + `model_name` + `field_name` as the key instead, but that makes it hard to support multiple translated strings in a complex field like `dashboard.json_metadata`.. maybe we need a dotted `sub_field` too, e.g. `native_filter_configuration.name`.
> * Currently, languages are switched via the menu, but there have been requests to respect locales provided by the browser, and allow the dropdown to be an override for that. Not sure if that fits into your vision, but it may be worth consideration
Sure, this proposal doesn't contradict adding support for browser-inferred language settings -- we just need to know what the current session's chosen language is, wherever it comes from. I looked into storing the user's selected language in the User preferences (https://github.com/openedx/openedx-aspects/issues/298), but abandoned that effort because it would require changing the FAB User model. (Ended up just [pulling it from the Open edX session](https://github.com/openedx/tutor-contrib-aspects/pull/1017), since our Superset logins are all via SSO.)
> * Others have brought up the idea that they may have JSON/YAML files that contain many of these strings. No need to worry about it here, but it does seem plausible to bolt on an "importer" that would plop such things into the table you're proposing.
Absolutely. Open edX needs to export/import translations from the command-line using `.mo/.po` files, so I included that in Phase 1 here. No reason why we couldn't support additional import formats as part of a later phase.
> * I appreciate that this is all UUID-based. I'm not sure which objects/models are not yet migrated to UUIDs as keys, but we there may be some prerequisite work there.
Interesting. So far we've only identified translatable fields in `dashboard`, `chart` and `dataset` (see Appendix of https://github.com/apache/superset/issues/32139), and so that was what I was planning to support initially.
> * Not sure if it needs to be considered here, but based on the locale (via browser, or menu selection) this ties into an existing wish list item of automagically converting number formatters to the desired locale, which can be done with vanilla javascript nowadays.
Oh yes, localization is bigger than just strings -- we need better number localization and RTL support too, but I think that should be done as a separate change. For now, I've just put them under "Out of Scope".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset data [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-2791172667
Found a seemingly abandoned but potentially relevant PR: https://github.com/apache/superset/pull/23252
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset data [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-2790188048
I think we're both pretty aligned on all this. Few specific comments
> I'm definitely in favor of a simple UI for these translations. But there's a huge advantage to having an in-context UI when humans are translating these strings, and I'd like to preserve that.
Right, context is everything. The way get_text framework keep tab of context is they include line-of-code pointers metadata along with the to-be-translated strings (optionally). Interestingly we decided to turn this off in our app-translation frawework as it operated like a merge-conflict-generating machine that was not manageable, now resorting to `grep` as the way to find entries. Since we're not parsing source code here, we have limited ways to identify context as the jinja code becomes some sort of runtime trigger. Unclear how much we can do from there (meaning jinja) either through interpreter-introspection or jinja-related self-awareness. Wouldn't be bad to have an extra field `context` in `i18n_translation`, it's probably in-scope for the SIP to define:
- whether we need/want that
- what the context fields looks like
- how it's instrumented (whether it's through introspection or explicitly defined by the caller, ie: `{{i18n('Translate this!', context='dashboard_title')}}` )
> since we're storing the asset UUID
Oh I think I had missed that from the spec, was thinking uuid was just a unique id for the translation object itself. Now if we have the uuid of the object itself there's a few thoughts related to that:
- can probably be introspected - probably at "entity-save-time", meaning when we save a dashboard, we trigger a routine that uses jinja to find all of the `i18n` calls in the various templated fields, and adds missing one to the translation table
- would couple nicely with `object_type` for a bit more referential integrity, though in theory we could figure this out somehow since uuids are globally unique. My take would be to have both `object_uuid` and `object_type` in `i18n_translation`, maybe `object_field` is useful here too, letting us know if it's used in `dashboard_title` or `dashboard_description` for instance. That allows for more context in the translation app and more specific retrieval when processing templated fields for retrieval
- building upon my previous command about the `context` field, maybe that's not needed anymore with the 2-3 fields I identified above. Though it could be nice to have an optional `extra_context` field too maybe to enrich the UX for the translation tool, as in `{{ i18n("Translate this!", extra_context="priority.p1") }}`
#### about garbage collection
Now if we have `object_uuid` and `object_type`, could even set up triggers on-delete and clean up orphaned entries in the translation table.
#### about reuse
Now if all framework is strict about capturing `object_uuid` and `object_type`, this means that multiple entries of the same string will have multiple entries to be translated. My guess is the framework would tend to not have a lot of duplicated instance of the same string, and that potentially you may want to translate it differently in different contexts (?) Maybe the translation app could offer a "propagate" feature, where it would look for identical strings across `object_uuid`s and duplicate based on existing translations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset metadata [superset]
Posted by "pomegranited (via GitHub)" <gi...@apache.org>.
pomegranited commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-3035380267
Hi @nieuwefn , unfortunately we were not able to secure funding to continue development here 👎
But our community still needs this, so I have hope that someone will be able to pick up the work started here. The discussions were valuable, and clarification of what Superset and Preset want from this feature was really useful.
So I'm going to close this issue, but I hope it will see some traction soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset metadata [superset]
Posted by "atstattus4 (via GitHub)" <gi...@apache.org>.
atstattus4 commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-4167538793
We need this feature a lot for our Superset integration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset metadata [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-3607954634
Does anyone else on the thread still want to take this on? If so, please let it be known, and we can consider a [VOTE], otherwise it may soon be marked as abandoned/discarded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset data [superset]
Posted by "pomegranited (via GitHub)" <gi...@apache.org>.
pomegranited commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-2788318466
Thank you for your feedback @mistercrunch !
> ### Clarifying the use of string_ids and/or use strings as keys in i18n_translations.
My thinking here was to avoid having these jinja filters in the actual fields, and wrap the entire string for templated fields at runtime.
This gives us a few advantages which I detailed in the final point under "Rejected Alternatives". What do you think?
> ### About caching / About batch-retrieval
Ah good point that batch-retrieval probably won't work for a jinja filter! I have added a `I18N_ASSET_TRANSLATIONS_CACHE_CONFIG` setting to the description above, so operators can configure caching if desired.
> ### About auto-populating translation through a service
> In the age of AI, I think no one should manually translate stuff anymore.
Agreed, but we also need to be able to correct what the AI gets wrong, so I don't think we can ditch the UI completely.
> The jinja macro, as it encounters new strings, will auto-populate rows in i18n_translations and flag them as "to-be-translated".
I don't think we need to flag them -- the absence of a translation for a given language is enough to know it needs to be translated.
> It seems writing a simple job, or generating async jobs to call a service using celery-beat (a cron that can run every N minutes that looks for new string, batches a call to GPT or google translate).
Someone can write this for sure -- but I won't do it as part of this SIP because of the open source issue. I am providing a command-line tool for extraction/importing translations though, so will ensure that API I create is general enough for someone to use it with a machine translation service.
> Even if we do want/need a UI for translation, i'd suggest building it externally to Superset maybe (similar to the poedit approach), so we'd only have to expose a CRUD REST API (if even) as you could slap a simple UI on top (Claude can probably build one quickly, or even plug a simple no-code solution like ReTool or AirTable on top of that table/model). It punts this UI complexity outside of Superset.
I'm definitely in favour of a simple UI for these translations. But there's a huge advantage to having an in-context UI when humans are translating these strings, and I'd like to preserve that.
> ### About garbage collection
> Lots of strings might get orphaned, and can be good to trim the table. That would require maintaining some sort of "last_used_dttm" thing, but that's more IO to manage.
I think we can trim the table without recording "last used" -- since we're storing the asset UUID + field and model names, we can easily identify translated strings that are no longer used. I've added a 2nd "command line tool" to handle this for Phase 1 -- operators can run/cron this as needed.
> ### About Jinja overhead
> ...simply not bothering with doing the template logic if/when we're in a non-i18n environment.
True.. I've updated the description here to state that this feature will be enabled/used only if the feature flag + multiple languages are enabled in Superset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset data [superset]
Posted by "pomegranited (via GitHub)" <gi...@apache.org>.
pomegranited commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-2814634976
@rusackas @mistercrunch
> Clearly, this is ready to bring into a [DISCUSS] thread on the dev@ mailing list. Sounds like ideas are starting to align here, so perhaps some open questions
Cool! I've sent one out with subject "[DISCUSS][SIP-161]".
I've pulled the larger open questions up into this task description -- please let me know if I've missed anything?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset metadata [superset]
Posted by "khashashin (via GitHub)" <gi...@apache.org>.
khashashin commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-3419840366
Very unfortunate this discussion didn’t continue, it looks like metadata i18n is no longer being actively pursued, which is a real shame for multi-language deployments.
I really like the TRANSLATION_HOOK approach mentioned above:
> From a deep conversation with PMCs today - there seemed to be traction for a simpler solution around a more generic "translation hook", where administrators would be free to implement a generic function to hook up with an external translation service. The high level idea is that you would have some sort of TRANSLATION_HOOK(text_to_be_translated, context_object) function defined in something like superset_config.py and externalize the translation process through that. That hook would assume the external service is in charge of accumulating strings to be translated, memoization, and serving the text -> do whatever makes sense to you in there.
That abstraction is exactly the kind of lightweight, pluggable interface that would let Superset remain minimal while enabling many deployment-specific workflows. In my case I need per-dashboard / per-chart multitenancy: each dashboard belongs to a tenant and the translation platform for each tenant is different. The `context_object` idea maps perfectly to that, the hook can receive tenant/dashboard identifiers and route to the correct external source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset metadata [superset]
Posted by "bmtcril (via GitHub)" <gi...@apache.org>.
bmtcril commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-3676115874
This SIP originally came out of our use case for Open edX, and we'd still love to see the work done but are in the same boat of not being able to push it forward ourselves.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset metadata [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-2898611915
I think we want to lean less on Flask AppBuilder (FAB). @mistercrunch is of the opinion this could be done with a Jinja macro to templatize these strings, and tap into translations stored wherever they may live. I'll let him provide info on extending the Jinja context to accommodate here.
We also considered extending the `t()` function that exists today as an extension point that can call APIs or do _whatever_ is needed to acquire a translation on the fly.
This might also be a use case for the forthcoming Extensions Architecture, but that would take some patience, as that's very much in the design/POC/concept phase at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset data [superset]
Posted by "jpchev (via GitHub)" <gi...@apache.org>.
jpchev commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-2771393107
@pomegranited as far as I understand the feature is to let dashboard creators provide a translation in multiple languages (localisation) for their content (chart title, descriptions....), right?
I don't need this for my projets, but I think it's very interesting and adding value to the Superset product.
I don't know what is open edx, though, is it a project you are working on?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP] Translating Superset asset data [superset]
Posted by "pomegranited (via GitHub)" <gi...@apache.org>.
pomegranited commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-2771108268
FYI @jpchev -- are you interested in this proposal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] [SIP-161] Translating Superset asset data [superset]
Posted by "graphwire (via GitHub)" <gi...@apache.org>.
graphwire commented on issue #32854:
URL: https://github.com/apache/superset/issues/32854#issuecomment-2774657178
Great proposal!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org