You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "AddouIsmail (via GitHub)" <gi...@apache.org> on 2023/03/01 17:59:48 UTC
[GitHub] [superset] AddouIsmail opened a new pull request, #23252: feat: Jinja template to translate contents dynamically
AddouIsmail opened a new pull request, #23252:
URL: https://github.com/apache/superset/pull/23252
<!---
Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/
Example:
fix(dashboard): load charts correctly
-->
### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
#### Problematic to solve:
To manage translation for dimensions, we need to create various calculated dimensions for each language.
If you want your dashboard to be translated in 5 languages, let's say because your organisation is present in 5 countries. So in this case you may create 5 calculated dimensions (for each column) that corresponds to the number of translations. And you need to create 1 dashboard / language using the appropriate translated column.
The total number of translation is the product of `distinct values count for all columns x languages`
#### Feature description
This PR introduce a new function to Jinja context that allows translation ability for features with the least effort.
The used library is `translate == 3.6.1`, it's quite simple as it calls api of translation services such as : `deepl`, `mymemory`, `microsoft` or also `libre translate`. This brings the ability to translate and have all in 1 dashboard and reduce number of calculated columns.
### BEFORE
to handle translations for SQL we have to handle it value by value, so we create as many custom dimensions as the number of languages that we want to handle
```SQL
SELECT
CASE
WHEN value = 'Good morning' THEN 'Bonjour'
WHEN value = 'Good evening' THEN 'Bonsoir'
ELSE 'No translation found'
END AS greeting_fr
```
### AFTER
Things could get much simpler, by creating 1 custom dimension or write it directly inside your dataset SQL query.
```SQL
SELECT {{translate(value, 'fr')}} AS greeting_fr
```
or
```SQL
SELECT {{translate(value, from_language='en', to_language='fr', add_to_cache_keys=True, escape_result=True)}} AS greeting_fr
```
### TESTING INSTRUCTIONS
1. Set `"ENABLE_TEMPLATE_PROCESSING": True` on the config
2. Optional: set `TRANSLATION_PROVIDER` to one of the available providers `deepl`, `mymemory`, `microsoft`, `libre`.
3. Test on SQL Lab or create your dynamically translated dimension
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [x] Introduces new feature or API
- [ ] Removes existing feature or API
This PR includes a fix of the error: `error: Unused "type: ignore" comment` that happens during the pre-commit.
### Some next steps doable after this PR:
If this PR is accepted as a feature for superset, there is some next step features that could be implemented, or created on the issue tab.
- **feature 1:** Create translation table that maps values to their various translations in other languages. That could be enriched through the interface or through file imports. Therefore the current `translate` template will mainly use the translation mappings if exists and fallback on the api if the translation is not found.
- **feature 2**: Link the translation to the selected language on Superset
- **feature 3**: Send the translation language on the body of the guest_token for embedding
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Jinja template to translate contents dynamically [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas closed pull request #23252: feat: Jinja template to translate contents dynamically
URL: https://github.com/apache/superset/pull/23252
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] dpgaspar commented on a diff in pull request #23252: feat: Jinja template to translate contents dynamically
Posted by "dpgaspar (via GitHub)" <gi...@apache.org>.
dpgaspar commented on code in PR #23252:
URL: https://github.com/apache/superset/pull/23252#discussion_r1133639563
##########
setup.py:
##########
@@ -126,6 +126,7 @@ def get_git_sha() -> str:
"wtforms>=2.3.3, <2.4",
"wtforms-json",
"xlsxwriter>=3.0.7, <3.1",
+ "translate == 3.6.1",
Review Comment:
this could be made an extra requirement
##########
superset/jinja_context.py:
##########
@@ -344,6 +346,54 @@ def get_filters(self, column: str, remove_filter: bool = False) -> List[Filter]:
return filters
+ def translate(
+ self,
+ string_to_translate: str,
+ to_language: str,
+ from_language: str = "en",
+ add_to_cache_keys: bool = True,
+ escape_result: bool = True,
+ ) -> str:
+ """
+ To get a translation dynamically through Jinja templating
+
+ Usage example: (Translation to french)
+ SELECT '{{ translate('my sentence', 'fr') }}' AS translation
+
+ Output:
+ translation
+ -----------
+ ma phrase
+
+ Python package : https://pypi.org/project/translate/
+ Available languages:
+
+ https://en.wikipedia.org/wiki/ISO_639-1
+ Examples: (e.g. en, ja, ko, pt, zh, zh-TW, ...)
+ """
+ if to_language is None:
+ return string_to_translate
+
+ provider_name = current_app.config.get("PROVIDER_CREDENTIALS_DICT", "mymemory")
+ secret = current_app.config.get("TRANSLATION_SECRET_ACCESS_KEY", None)
Review Comment:
We assume on the code base that all the necessary config keys are always present with a default value. On this case it could make sense to just rely on the package config methodology. More if we decide to use this package has an extra.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Jinja template to translate contents dynamically [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #23252:
URL: https://github.com/apache/superset/pull/23252#issuecomment-2822264118
This still hasn't been touched since December, so I'll go ahead and close it. Please feel free to reopen if warranted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Jinja template to translate contents dynamically [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #23252:
URL: https://github.com/apache/superset/pull/23252#issuecomment-1936538159
Hi @AddouIsmail - any intent to follow up on this PR? Just making sure questions get addressed, and that the PR gets a rebase if you want to merge it. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Jinja template to translate contents dynamically [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #23252:
URL: https://github.com/apache/superset/pull/23252#issuecomment-2529037310
Converting this to draft, since it still needs a rebase. It might be closed in time if it doesn't get any attention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] commented on pull request #23252: feat: Jinja template to translate contents dynamically
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #23252:
URL: https://github.com/apache/superset/pull/23252#issuecomment-1451917567
# [Codecov](https://codecov.io/gh/apache/superset/pull/23252?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#23252](https://codecov.io/gh/apache/superset/pull/23252?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8c4d9da) into [master](https://codecov.io/gh/apache/superset/commit/9a4839f45c6d9a2cae89f5c77b434910be129f48?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9a4839f) will **decrease** coverage by `1.77%`.
> The diff coverage is `59.19%`.
> :exclamation: Current head 8c4d9da differs from pull request most recent head 9746e29. Consider uploading reports for the commit 9746e29 to get more accurate results
```diff
@@ Coverage Diff @@
## master #23252 +/- ##
==========================================
- Coverage 67.50% 65.74% -1.77%
==========================================
Files 1898 1907 +9
Lines 72978 73445 +467
Branches 7899 7973 +74
==========================================
- Hits 49262 48283 -979
- Misses 21691 23114 +1423
- Partials 2025 2048 +23
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `?` | |
| mysql | `78.40% <69.70%> (-0.08%)` | :arrow_down: |
| postgres | `78.46% <69.70%> (-0.08%)` | :arrow_down: |
| presto | `?` | |
| python | `78.57% <69.70%> (-3.76%)` | :arrow_down: |
| sqlite | `76.93% <68.07%> (-0.08%)` | :arrow_down: |
| unit | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/superset/pull/23252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...set-frontend/src/components/Table/VirtualTable.tsx](https://codecov.io/gh/apache/superset/pull/23252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2NvbXBvbmVudHMvVGFibGUvVmlydHVhbFRhYmxlLnRzeA==) | `72.30% <0.00%> (-1.13%)` | :arrow_down: |
| [...t-frontend/src/dashboard/actions/dashboardState.js](https://codecov.io/gh/apache/superset/pull/23252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2Rhc2hib2FyZC9hY3Rpb25zL2Rhc2hib2FyZFN0YXRlLmpz) | `55.76% <0.00%> (+1.77%)` | :arrow_up: |
| [superset-frontend/src/dashboard/actions/hydrate.js](https://codecov.io/gh/apache/superset/pull/23252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2Rhc2hib2FyZC9hY3Rpb25zL2h5ZHJhdGUuanM=) | `2.06% <0.00%> (+0.19%)` | :arrow_up: |
| [...ard/components/FiltersBadge/DetailsPanel/index.tsx](https://codecov.io/gh/apache/superset/pull/23252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2Rhc2hib2FyZC9jb21wb25lbnRzL0ZpbHRlcnNCYWRnZS9EZXRhaWxzUGFuZWwvaW5kZXgudHN4) | `82.22% <ø> (ø)` | |
| [.../components/FiltersBadge/FilterIndicator/index.tsx](https://codecov.io/gh/apache/superset/pull/23252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2Rhc2hib2FyZC9jb21wb25lbnRzL0ZpbHRlcnNCYWRnZS9GaWx0ZXJJbmRpY2F0b3IvaW5kZXgudHN4) | `87.50% <ø> (ø)` | |
| [...nd/src/dashboard/components/FiltersBadge/index.tsx](https://codecov.io/gh/apache/superset/pull/23252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2Rhc2hib2FyZC9jb21wb25lbnRzL0ZpbHRlcnNCYWRnZS9pbmRleC50c3g=) | `86.11% <0.00%> (ø)` | |
| [...Filters/FilterBar/FilterControls/FilterControl.tsx](https://codecov.io/gh/apache/superset/pull/23252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2Rhc2hib2FyZC9jb21wb25lbnRzL25hdGl2ZUZpbHRlcnMvRmlsdGVyQmFyL0ZpbHRlckNvbnRyb2xzL0ZpbHRlckNvbnRyb2wudHN4) | `28.57% <ø> (ø)` | |
| [...veFilters/FilterBar/FilterControls/FilterValue.tsx](https://codecov.io/gh/apache/superset/pull/23252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2Rhc2hib2FyZC9jb21wb25lbnRzL25hdGl2ZUZpbHRlcnMvRmlsdGVyQmFyL0ZpbHRlckNvbnRyb2xzL0ZpbHRlclZhbHVlLnRzeA==) | `5.71% <0.00%> (-0.29%)` | :arrow_down: |
| [...Filters/FilterBar/FiltersDropdownContent/index.tsx](https://codecov.io/gh/apache/superset/pull/23252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2Rhc2hib2FyZC9jb21wb25lbnRzL25hdGl2ZUZpbHRlcnMvRmlsdGVyQmFyL0ZpbHRlcnNEcm9wZG93bkNvbnRlbnQvaW5kZXgudHN4) | `20.00% <0.00%> (-5.00%)` | :arrow_down: |
| [...s/FilterBar/FiltersOutOfScopeCollapsible/index.tsx](https://codecov.io/gh/apache/superset/pull/23252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2Rhc2hib2FyZC9jb21wb25lbnRzL25hdGl2ZUZpbHRlcnMvRmlsdGVyQmFyL0ZpbHRlcnNPdXRPZlNjb3BlQ29sbGFwc2libGUvaW5kZXgudHN4) | `16.66% <0.00%> (-3.34%)` | :arrow_down: |
| ... and [186 more](https://codecov.io/gh/apache/superset/pull/23252?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org