You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "rusackas (via GitHub)" <gi...@apache.org> on 2026/04/17 17:04:56 UTC
[PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
rusackas opened a new pull request, #39448:
URL: https://github.com/apache/superset/pull/39448
### SUMMARY
Adds two scripts to help maintainers fill in missing `.po` translations using Claude AI as a draft generator, then applies them to backfill all 184 missing Spanish strings.
**New scripts:**
- `scripts/translations/build_translation_index.py` — reads every `.po` file and outputs a cross-language JSON index (`{msgid: {lang: translation}}`) used as reference context for the AI
- `scripts/translations/backfill_po.py` — for a target language, finds all untranslated entries, batches them (50/request), and calls `claude -p` with cross-language context to generate draft translations
**Design highlights:**
- Cross-language translations are passed per-string so the AI can disambiguate ambiguous English terms (e.g. "Scale", "Turkey", "Table") from how translators in other languages have already resolved the same string
- `--min-context N` flag skips strings with fewer than N reference translations in other languages — useful for excluding brand-new strings with no disambiguation signal
- Each AI-generated entry gets a translator comment with the model used and which languages provided context (e.g. `# Machine-translated via backfill_po.py (claude-sonnet-4-6) [refs: fr, ru]`)
- All generated entries are marked `#, fuzzy` so they surface in the standard translation review workflow
- `translation_index.json` added to `.gitignore` (it's a local build artifact)
**Spanish translations (`es`):**
- Backfilled all 184 previously untranslated strings
- All entries marked `#, fuzzy` pending native-speaker review
**Docs:** Added a "Backfilling missing translations with AI" section to `docs/developer_docs/contributing/howtos.md` with step-by-step instructions and a full options reference table.
**npm shortcuts** in `superset-frontend/package.json`:
- `npm run translations:build-index`
- `npm run translations:backfill -- --lang fr`
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
N/A — this is tooling and `.po` file changes only.
### TESTING INSTRUCTIONS
1. Install deps: `pip install -r superset/translations/requirements.txt`
2. Ensure `claude --version` works (Claude Code must be authenticated)
3. Build the index: `python scripts/translations/build_translation_index.py`
4. Dry-run for a language: `python scripts/translations/backfill_po.py --lang de --limit 10 --dry-run`
5. Verify output shows translated strings with `[refs: ...]` context tags
6. Inspect `superset/translations/es/LC_MESSAGES/messages.po` — all previously empty `msgstr` fields should now be populated and marked `#, fuzzy`
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration
- [x] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #39448:
URL: https://github.com/apache/superset/pull/39448#issuecomment-4521593767
> A non-blocking question is about hard-coding this to use Claude Code. I'm fine with that since it's not a non-core part of the repo, but ideally it would be configured to work with other LLM subscriptions
1000% agreed. And, actually, we won't get away with this for long, as "programatic use" on Claude Code is about to get _expensive_ starting June 15th. So I fully intend to do _some sort_ of follow up here, but perhaps we can translate the world before that date if this merges.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #39448:
URL: https://github.com/apache/superset/pull/39448#issuecomment-4502796130
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #39448:
URL: https://github.com/apache/superset/pull/39448#discussion_r3105993525
##########
scripts/translations/backfill_po.py:
##########
@@ -0,0 +1,557 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Backfill missing translations in a .po file using Claude AI.
+
+For each untranslated (empty msgstr) entry in the target language, the script
+sends the English source string along with all available translations in other
+languages to Claude as context, then writes the AI-generated translation back
+into the .po file marked as #, fuzzy for human review.
+
+Usage:
+ # Build the translation index first (one-time or when .po files change)
+ python scripts/translations/build_translation_index.py
+
+ # Backfill French translations
+ python scripts/translations/backfill_po.py --lang fr
+
+ # Dry run (print what would be translated without writing)
+ python scripts/translations/backfill_po.py --lang de --dry-run
+
+ # Limit to 100 entries and use a specific model
+ python scripts/translations/backfill_po.py --lang es --limit 100 \
+ --model claude-opus-4-6
+
+Options:
+ --lang LANG ISO language code to backfill (required)
+ --batch-size N Number of strings per Claude request (default: 50)
+ --limit N Stop after translating N entries (default: unlimited)
+ --model MODEL Claude model ID (default: claude-sonnet-4-6)
+ --index PATH Path to translation_index.json (default: auto-detect)
+ --dry-run Print translations without writing to .po file
+ --fuzzy/--no-fuzzy Mark generated translations as fuzzy (default: fuzzy)
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any
+
+try:
+ import polib # type: ignore[import-untyped]
+except ImportError:
+ print("polib is required. Run: pip install polib", file=sys.stderr)
+ sys.exit(1)
+
+TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations"
+DEFAULT_INDEX = TRANSLATIONS_DIR / "translation_index.json"
+DEFAULT_MODEL = "claude-sonnet-4-6"
+DEFAULT_BATCH_SIZE = 50
+
+# Language names for the prompt, keyed by ISO code
+LANGUAGE_NAMES: dict[str, str] = {
+ "ar": "Arabic",
+ "ca": "Catalan",
+ "de": "German",
+ "es": "Spanish",
+ "fa": "Persian (Farsi)",
+ "fr": "French",
+ "it": "Italian",
+ "ja": "Japanese",
+ "ko": "Korean",
+ "mi": "Māori",
+ "nl": "Dutch",
+ "pl": "Polish",
+ "pt": "Portuguese",
+ "pt_BR": "Brazilian Portuguese",
+ "ru": "Russian",
+ "sk": "Slovak",
+ "sl": "Slovenian",
+ "tr": "Turkish",
+ "uk": "Ukrainian",
+ "zh": "Chinese (Simplified)",
+ "zh_TW": "Chinese (Traditional)",
+}
+
+
+def _lang_name(code: str) -> str:
+ return LANGUAGE_NAMES.get(code, code)
+
+
+def _plural_key(msgid: str, msgid_plural: str) -> str:
+ return f"{msgid}\x00{msgid_plural}"
+
+
+def _is_missing(entry: polib.POEntry) -> bool:
+ """Return True for entries that need a translation."""
+ if entry.obsolete:
+ return False
+ if entry.msgid_plural:
+ return not any(v for v in entry.msgstr_plural.values())
+ return not entry.msgstr
+
+
+def _context_langs(
+ item: dict[str, Any], index: dict[str, Any], target_lang: str
+) -> list[str]:
+ """Return sorted list of language codes that have translations for this entry."""
+ key = item["index_key"]
+ if key not in index:
+ return []
+ return sorted(
+ lang for lang, val in index[key].items() if lang != target_lang and val
+ )
+
+
+def _context_count(
+ item: dict[str, Any], index: dict[str, Any], target_lang: str
+) -> int:
+ """Return the number of other-language translations available for this entry."""
+ return len(_context_langs(item, index, target_lang))
+
+
+def _render_item(
+ i: int,
+ item: dict[str, Any],
+ index: dict[str, Any],
+ target_lang: str,
+ reference_langs_sorted: list[str],
+) -> list[str]:
+ """Render one batch entry as prompt lines."""
+ lines: list[str] = []
+ ctx = _context_count(item, index, target_lang)
+ if ctx == 0:
+ lines.append(
+ f"--- [{i}] (no reference translations — translate conservatively) ---"
+ )
+ else:
+ plural = "s" if ctx != 1 else ""
+ lines.append(f"--- [{i}] ({ctx} reference translation{plural}) ---")
+ lines.append(f"English: {json.dumps(item['msgid'], ensure_ascii=False)}")
+ if item.get("msgid_plural"):
+ plural_json = json.dumps(item["msgid_plural"], ensure_ascii=False)
+ lines.append(f"English plural: {plural_json}")
+ key = item["index_key"]
+ if key in index and reference_langs_sorted:
+ for lang in reference_langs_sorted:
+ val = index[key].get(lang)
+ if val is None:
+ continue
+ if isinstance(val, dict):
+ forms = "; ".join(
+ f"[{k}] {json.dumps(v, ensure_ascii=False)}" for k, v in val.items()
+ )
+ lines.append(f"{_lang_name(lang)}: {forms}")
+ else:
+ lines.append(
+ f"{_lang_name(lang)}: {json.dumps(val, ensure_ascii=False)}"
+ )
+ lines.append("")
+ return lines
+
+
+def build_prompt(
+ target_lang: str,
+ batch: list[dict[str, Any]],
+ index: dict[str, Any],
+) -> str:
+ """Build the Claude prompt for a batch of entries."""
+ lang_name = _lang_name(target_lang)
+
+ # Collect which other languages actually have translations for this batch
+ reference_langs: set[str] = set()
+ for item in batch:
+ key = item["index_key"]
+ if key in index:
+ reference_langs.update(
+ lang for lang, val in index[key].items() if lang != target_lang and val
+ )
+ reference_langs_sorted = sorted(reference_langs)
+
+ lines: list[str] = [
+ "You are a professional translator specializing in software UI strings.",
+ f"Translate the following English strings into {lang_name} ({target_lang}).",
+ "",
+ "Rules:",
+ "- Preserve all format placeholders exactly: %(name)s, {name}, %s, %d, etc.",
+ "- Preserve HTML tags if present.",
+ "- Keep the same tone and register as the reference translations.",
+ "- For plural forms, provide translations for all plural forms"
+ " required by the language.",
+ "- Return ONLY a JSON object mapping each numeric index (as a string)"
+ " to its translation.",
+ "- Do not add any explanation, preamble, or markdown fences.",
+ "",
+ "Important: Many strings are short fragments or single words that are"
+ " ambiguous in English (e.g. 'Scale' could mean a measurement scale,"
+ " to scale an image, or fish scales). Use the translations in other"
+ " languages as your primary signal for which meaning is intended —"
+ " they collectively disambiguate the intended sense. When no"
+ " other-language translations are available for an entry, translate"
+ " conservatively based on the most common meaning in a data"
+ " visualization UI context.",
+ "",
+ ]
+
+ if reference_langs_sorted:
+ lines.append(
+ f"Reference translations are provided per string where available "
+ f"({', '.join(_lang_name(lc) for lc in reference_langs_sorted)})."
+ )
+ lines.append("")
+
+ lines.append("Strings to translate:")
+ lines.append("")
+
+ for i, item in enumerate(batch):
+ lines.extend(_render_item(i, item, index, target_lang, reference_langs_sorted))
+
+ if batch and batch[0].get("msgid_plural"):
+ # Add guidance on plural form counts per language
+ lines.append(
+ "Note: provide ALL plural forms required by the target language "
+ "(e.g. French needs 2, Russian needs 3, Arabic needs 6)."
+ )
+ lines.append("")
+
+ lines.append(
+ 'Expected output format: {"0": "<translation>", "1": "<translation>", ...}'
+ )
+ lines.append("(keys are the numeric indices of the strings above)")
+
+ return "\n".join(lines)
+
+
+def parse_response(text: str, batch_size: int) -> dict[int, str]:
+ """Parse the JSON object from Claude's response."""
+ # Strip any accidental markdown fences
+ text = re.sub(r"^```[^\n]*\n", "", text.strip())
+ text = re.sub(r"\n```$", "", text)
+ try:
+ raw = json.loads(text)
+ return {int(k): str(v) for k, v in raw.items() if str(k).isdigit()}
+ except json.JSONDecodeError as exc:
+ raise ValueError(
+ f"Could not parse response as JSON: {exc}\n\nResponse:\n{text}"
+ ) from exc
+
+
+def translate_batch(
+ model: str,
+ target_lang: str,
+ batch: list[dict[str, Any]],
+ index: dict[str, Any],
+) -> dict[int, str]:
+ """Send a batch of strings to Claude via `claude -p`.
+
+ Returns a dict mapping batch index to translated string.
+ """
+ claude_bin = shutil.which("claude")
+ if not claude_bin:
+ raise RuntimeError(
+ "claude CLI not found. Install Claude Code or add it to PATH."
+ )
+ prompt = build_prompt(target_lang, batch, index)
+ # claude_bin is resolved via shutil.which — not user-controlled input
+ result = subprocess.run( # noqa: S603
+ [claude_bin, "--model", model, "-p", prompt],
+ capture_output=True,
+ text=True,
+ check=False,
+ )
+ if result.returncode != 0:
+ raise RuntimeError(
+ f"claude exited with code {result.returncode}:\n{result.stderr}"
+ )
+ return parse_response(result.stdout.strip(), len(batch))
+
+
+def _apply_translation(
+ entry: polib.POEntry,
+ translation: str,
+ item: dict[str, Any],
+ model: str,
+ mark_fuzzy: bool,
+) -> None:
+ """Write a translation string into a POEntry and add attribution."""
+ if entry.msgid_plural:
+ # Model may return a JSON dict of plural forms or a plain string
+ try:
+ plural_dict = json.loads(translation)
+ if isinstance(plural_dict, dict):
+ entry.msgstr_plural = {int(k): str(v) for k, v in plural_dict.items()}
+ else:
+ for k in entry.msgstr_plural:
+ entry.msgstr_plural[k] = str(plural_dict)
+ except (json.JSONDecodeError, ValueError):
+ for k in entry.msgstr_plural:
+ entry.msgstr_plural[k] = translation
+ else:
+ entry.msgstr = translation
+
+ if mark_fuzzy and "fuzzy" not in entry.flags:
+ entry.flags.append("fuzzy")
+
+ refs = item["context_langs"]
+ refs_tag = f" [refs: {', '.join(refs)}]" if refs else " [no refs]"
+ attribution = f"Machine-translated via backfill_po.py ({model}){refs_tag}"
+ if entry.tcomment:
+ if attribution not in entry.tcomment:
+ entry.tcomment = f"{entry.tcomment}\n{attribution}"
+ else:
+ entry.tcomment = attribution
+
+
+def _build_batch_items(
+ entries: list[polib.POEntry],
+ index: dict[str, Any],
+ lang: str,
+) -> list[dict[str, Any]]:
+ """Convert a list of POEntries into the dict format used by translate_batch."""
+ items: list[dict[str, Any]] = []
+ for entry in entries:
+ if entry.msgid_plural:
+ item: dict[str, Any] = {
+ "msgid": entry.msgid,
+ "msgid_plural": entry.msgid_plural,
+ "index_key": _plural_key(entry.msgid, entry.msgid_plural),
+ "is_plural": True,
+ }
+ else:
+ item = {
+ "msgid": entry.msgid,
+ "index_key": entry.msgid,
+ "is_plural": False,
+ }
+ item["context_langs"] = _context_langs(item, index, lang)
+ item["context_count"] = len(item["context_langs"])
+ items.append(item)
+ return items
+
+
+def _process_batches(
+ missing: list[polib.POEntry],
+ index: dict[str, Any],
+ lang: str,
+ batch_size: int,
+ model: str,
+ dry_run: bool,
+ mark_fuzzy: bool,
+) -> tuple[int, int]:
+ """Translate missing entries in batches. Returns (translated, failed) counts."""
+ translated_count = 0
+ failed_count = 0
+ for batch_start in range(0, len(missing), batch_size):
+ batch_entries = missing[batch_start : batch_start + batch_size]
+ batch_items = _build_batch_items(batch_entries, index, lang)
+ end = min(batch_start + batch_size, len(missing))
+ print(
+ f" Translating entries {batch_start + 1}–{end} of {len(missing)} …",
+ file=sys.stderr,
+ )
+ try:
+ translations = translate_batch(model, lang, batch_items, index)
+ except (ValueError, RuntimeError) as exc:
+ print(f" ERROR in batch starting at {batch_start}: {exc}", file=sys.stderr)
+ failed_count += len(batch_entries)
+ continue
+ for i, entry in enumerate(batch_entries):
+ translation = translations.get(i)
+ if translation is None:
+ print(
+ f" WARNING: no translation returned for index {i} "
+ f"(msgid: {entry.msgid[:60]!r})",
+ file=sys.stderr,
+ )
+ failed_count += 1
+ continue
+ if dry_run:
+ ctx = batch_items[i]["context_count"]
+ ctx_tag = f" [ctx:{ctx}]" if ctx < 3 else ""
+ print(
+ f" [{lang}]{ctx_tag} {entry.msgid[:60]!r} → {translation[:60]!r}"
+ )
+ else:
+ _apply_translation(
+ entry, translation, batch_items[i], model, mark_fuzzy
+ )
+ translated_count += 1
+ return translated_count, failed_count
+
+
+def backfill(
+ lang: str,
+ *,
+ batch_size: int = DEFAULT_BATCH_SIZE,
+ limit: int | None = None,
+ min_context: int = 0,
+ model: str = DEFAULT_MODEL,
+ index_path: Path = DEFAULT_INDEX,
+ dry_run: bool = False,
+ mark_fuzzy: bool = True,
+) -> None:
Review Comment:
Correction to my earlier reply: this function genuinely didn't have a docstring applied yet (unlike the `_lang_name`/`_plural_key` cases where the suggestion replaced the body). Added the docstring in 1ac81f7a31.
##########
scripts/translations/backfill_po.py:
##########
@@ -0,0 +1,557 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Backfill missing translations in a .po file using Claude AI.
+
+For each untranslated (empty msgstr) entry in the target language, the script
+sends the English source string along with all available translations in other
+languages to Claude as context, then writes the AI-generated translation back
+into the .po file marked as #, fuzzy for human review.
+
+Usage:
+ # Build the translation index first (one-time or when .po files change)
+ python scripts/translations/build_translation_index.py
+
+ # Backfill French translations
+ python scripts/translations/backfill_po.py --lang fr
+
+ # Dry run (print what would be translated without writing)
+ python scripts/translations/backfill_po.py --lang de --dry-run
+
+ # Limit to 100 entries and use a specific model
+ python scripts/translations/backfill_po.py --lang es --limit 100 \
+ --model claude-opus-4-6
+
+Options:
+ --lang LANG ISO language code to backfill (required)
+ --batch-size N Number of strings per Claude request (default: 50)
+ --limit N Stop after translating N entries (default: unlimited)
+ --model MODEL Claude model ID (default: claude-sonnet-4-6)
+ --index PATH Path to translation_index.json (default: auto-detect)
+ --dry-run Print translations without writing to .po file
+ --fuzzy/--no-fuzzy Mark generated translations as fuzzy (default: fuzzy)
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any
+
+try:
+ import polib # type: ignore[import-untyped]
+except ImportError:
+ print("polib is required. Run: pip install polib", file=sys.stderr)
+ sys.exit(1)
+
+TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations"
+DEFAULT_INDEX = TRANSLATIONS_DIR / "translation_index.json"
+DEFAULT_MODEL = "claude-sonnet-4-6"
+DEFAULT_BATCH_SIZE = 50
+
+# Language names for the prompt, keyed by ISO code
+LANGUAGE_NAMES: dict[str, str] = {
+ "ar": "Arabic",
+ "ca": "Catalan",
+ "de": "German",
+ "es": "Spanish",
+ "fa": "Persian (Farsi)",
+ "fr": "French",
+ "it": "Italian",
+ "ja": "Japanese",
+ "ko": "Korean",
+ "mi": "Māori",
+ "nl": "Dutch",
+ "pl": "Polish",
+ "pt": "Portuguese",
+ "pt_BR": "Brazilian Portuguese",
+ "ru": "Russian",
+ "sk": "Slovak",
+ "sl": "Slovenian",
+ "tr": "Turkish",
+ "uk": "Ukrainian",
+ "zh": "Chinese (Simplified)",
+ "zh_TW": "Chinese (Traditional)",
+}
+
+
+def _lang_name(code: str) -> str:
+ return LANGUAGE_NAMES.get(code, code)
+
+
+def _plural_key(msgid: str, msgid_plural: str) -> str:
+ return f"{msgid}\x00{msgid_plural}"
+
+
+def _is_missing(entry: polib.POEntry) -> bool:
+ """Return True for entries that need a translation."""
+ if entry.obsolete:
+ return False
+ if entry.msgid_plural:
+ return not any(v for v in entry.msgstr_plural.values())
+ return not entry.msgstr
+
+
+def _context_langs(
+ item: dict[str, Any], index: dict[str, Any], target_lang: str
+) -> list[str]:
+ """Return sorted list of language codes that have translations for this entry."""
+ key = item["index_key"]
+ if key not in index:
+ return []
+ return sorted(
+ lang for lang, val in index[key].items() if lang != target_lang and val
+ )
+
+
+def _context_count(
+ item: dict[str, Any], index: dict[str, Any], target_lang: str
+) -> int:
+ """Return the number of other-language translations available for this entry."""
+ return len(_context_langs(item, index, target_lang))
+
+
+def _render_item(
+ i: int,
+ item: dict[str, Any],
+ index: dict[str, Any],
+ target_lang: str,
+ reference_langs_sorted: list[str],
+) -> list[str]:
+ """Render one batch entry as prompt lines."""
+ lines: list[str] = []
+ ctx = _context_count(item, index, target_lang)
+ if ctx == 0:
+ lines.append(
+ f"--- [{i}] (no reference translations — translate conservatively) ---"
+ )
+ else:
+ plural = "s" if ctx != 1 else ""
+ lines.append(f"--- [{i}] ({ctx} reference translation{plural}) ---")
+ lines.append(f"English: {json.dumps(item['msgid'], ensure_ascii=False)}")
+ if item.get("msgid_plural"):
+ plural_json = json.dumps(item["msgid_plural"], ensure_ascii=False)
+ lines.append(f"English plural: {plural_json}")
+ key = item["index_key"]
+ if key in index and reference_langs_sorted:
+ for lang in reference_langs_sorted:
+ val = index[key].get(lang)
+ if val is None:
+ continue
+ if isinstance(val, dict):
+ forms = "; ".join(
+ f"[{k}] {json.dumps(v, ensure_ascii=False)}" for k, v in val.items()
+ )
+ lines.append(f"{_lang_name(lang)}: {forms}")
+ else:
+ lines.append(
+ f"{_lang_name(lang)}: {json.dumps(val, ensure_ascii=False)}"
+ )
+ lines.append("")
+ return lines
+
+
+def build_prompt(
+ target_lang: str,
+ batch: list[dict[str, Any]],
+ index: dict[str, Any],
+) -> str:
+ """Build the Claude prompt for a batch of entries."""
+ lang_name = _lang_name(target_lang)
+
+ # Collect which other languages actually have translations for this batch
+ reference_langs: set[str] = set()
+ for item in batch:
+ key = item["index_key"]
+ if key in index:
+ reference_langs.update(
+ lang for lang, val in index[key].items() if lang != target_lang and val
+ )
+ reference_langs_sorted = sorted(reference_langs)
+
+ lines: list[str] = [
+ "You are a professional translator specializing in software UI strings.",
+ f"Translate the following English strings into {lang_name} ({target_lang}).",
+ "",
+ "Rules:",
+ "- Preserve all format placeholders exactly: %(name)s, {name}, %s, %d, etc.",
+ "- Preserve HTML tags if present.",
+ "- Keep the same tone and register as the reference translations.",
+ "- For plural forms, provide translations for all plural forms"
+ " required by the language.",
+ "- Return ONLY a JSON object mapping each numeric index (as a string)"
+ " to its translation.",
+ "- Do not add any explanation, preamble, or markdown fences.",
+ "",
+ "Important: Many strings are short fragments or single words that are"
+ " ambiguous in English (e.g. 'Scale' could mean a measurement scale,"
+ " to scale an image, or fish scales). Use the translations in other"
+ " languages as your primary signal for which meaning is intended —"
+ " they collectively disambiguate the intended sense. When no"
+ " other-language translations are available for an entry, translate"
+ " conservatively based on the most common meaning in a data"
+ " visualization UI context.",
+ "",
+ ]
+
+ if reference_langs_sorted:
+ lines.append(
+ f"Reference translations are provided per string where available "
+ f"({', '.join(_lang_name(lc) for lc in reference_langs_sorted)})."
+ )
+ lines.append("")
+
+ lines.append("Strings to translate:")
+ lines.append("")
+
+ for i, item in enumerate(batch):
+ lines.extend(_render_item(i, item, index, target_lang, reference_langs_sorted))
+
+ if batch and batch[0].get("msgid_plural"):
+ # Add guidance on plural form counts per language
+ lines.append(
+ "Note: provide ALL plural forms required by the target language "
+ "(e.g. French needs 2, Russian needs 3, Arabic needs 6)."
+ )
+ lines.append("")
+
+ lines.append(
+ 'Expected output format: {"0": "<translation>", "1": "<translation>", ...}'
+ )
+ lines.append("(keys are the numeric indices of the strings above)")
+
+ return "\n".join(lines)
+
+
+def parse_response(text: str, batch_size: int) -> dict[int, str]:
+ """Parse the JSON object from Claude's response."""
+ # Strip any accidental markdown fences
+ text = re.sub(r"^```[^\n]*\n", "", text.strip())
+ text = re.sub(r"\n```$", "", text)
+ try:
+ raw = json.loads(text)
+ return {int(k): str(v) for k, v in raw.items() if str(k).isdigit()}
+ except json.JSONDecodeError as exc:
+ raise ValueError(
+ f"Could not parse response as JSON: {exc}\n\nResponse:\n{text}"
+ ) from exc
+
+
+def translate_batch(
+ model: str,
+ target_lang: str,
+ batch: list[dict[str, Any]],
+ index: dict[str, Any],
+) -> dict[int, str]:
+ """Send a batch of strings to Claude via `claude -p`.
+
+ Returns a dict mapping batch index to translated string.
+ """
+ claude_bin = shutil.which("claude")
+ if not claude_bin:
+ raise RuntimeError(
+ "claude CLI not found. Install Claude Code or add it to PATH."
+ )
+ prompt = build_prompt(target_lang, batch, index)
+ # claude_bin is resolved via shutil.which — not user-controlled input
+ result = subprocess.run( # noqa: S603
+ [claude_bin, "--model", model, "-p", prompt],
+ capture_output=True,
+ text=True,
+ check=False,
+ )
+ if result.returncode != 0:
+ raise RuntimeError(
+ f"claude exited with code {result.returncode}:\n{result.stderr}"
+ )
+ return parse_response(result.stdout.strip(), len(batch))
+
+
+def _apply_translation(
+ entry: polib.POEntry,
+ translation: str,
+ item: dict[str, Any],
+ model: str,
+ mark_fuzzy: bool,
+) -> None:
+ """Write a translation string into a POEntry and add attribution."""
+ if entry.msgid_plural:
+ # Model may return a JSON dict of plural forms or a plain string
+ try:
+ plural_dict = json.loads(translation)
+ if isinstance(plural_dict, dict):
+ entry.msgstr_plural = {int(k): str(v) for k, v in plural_dict.items()}
+ else:
+ for k in entry.msgstr_plural:
+ entry.msgstr_plural[k] = str(plural_dict)
+ except (json.JSONDecodeError, ValueError):
+ for k in entry.msgstr_plural:
+ entry.msgstr_plural[k] = translation
+ else:
+ entry.msgstr = translation
+
+ if mark_fuzzy and "fuzzy" not in entry.flags:
+ entry.flags.append("fuzzy")
+
+ refs = item["context_langs"]
+ refs_tag = f" [refs: {', '.join(refs)}]" if refs else " [no refs]"
+ attribution = f"Machine-translated via backfill_po.py ({model}){refs_tag}"
+ if entry.tcomment:
+ if attribution not in entry.tcomment:
+ entry.tcomment = f"{entry.tcomment}\n{attribution}"
+ else:
+ entry.tcomment = attribution
+
+
+def _build_batch_items(
+ entries: list[polib.POEntry],
+ index: dict[str, Any],
+ lang: str,
+) -> list[dict[str, Any]]:
+ """Convert a list of POEntries into the dict format used by translate_batch."""
+ items: list[dict[str, Any]] = []
+ for entry in entries:
+ if entry.msgid_plural:
+ item: dict[str, Any] = {
+ "msgid": entry.msgid,
+ "msgid_plural": entry.msgid_plural,
+ "index_key": _plural_key(entry.msgid, entry.msgid_plural),
+ "is_plural": True,
+ }
+ else:
+ item = {
+ "msgid": entry.msgid,
+ "index_key": entry.msgid,
+ "is_plural": False,
+ }
+ item["context_langs"] = _context_langs(item, index, lang)
+ item["context_count"] = len(item["context_langs"])
+ items.append(item)
+ return items
+
+
+def _process_batches(
+ missing: list[polib.POEntry],
+ index: dict[str, Any],
+ lang: str,
+ batch_size: int,
+ model: str,
+ dry_run: bool,
+ mark_fuzzy: bool,
+) -> tuple[int, int]:
+ """Translate missing entries in batches. Returns (translated, failed) counts."""
+ translated_count = 0
+ failed_count = 0
+ for batch_start in range(0, len(missing), batch_size):
+ batch_entries = missing[batch_start : batch_start + batch_size]
+ batch_items = _build_batch_items(batch_entries, index, lang)
+ end = min(batch_start + batch_size, len(missing))
+ print(
+ f" Translating entries {batch_start + 1}–{end} of {len(missing)} …",
+ file=sys.stderr,
+ )
+ try:
+ translations = translate_batch(model, lang, batch_items, index)
+ except (ValueError, RuntimeError) as exc:
+ print(f" ERROR in batch starting at {batch_start}: {exc}", file=sys.stderr)
+ failed_count += len(batch_entries)
+ continue
+ for i, entry in enumerate(batch_entries):
+ translation = translations.get(i)
+ if translation is None:
+ print(
+ f" WARNING: no translation returned for index {i} "
+ f"(msgid: {entry.msgid[:60]!r})",
+ file=sys.stderr,
+ )
+ failed_count += 1
+ continue
+ if dry_run:
+ ctx = batch_items[i]["context_count"]
+ ctx_tag = f" [ctx:{ctx}]" if ctx < 3 else ""
+ print(
+ f" [{lang}]{ctx_tag} {entry.msgid[:60]!r} → {translation[:60]!r}"
+ )
+ else:
+ _apply_translation(
+ entry, translation, batch_items[i], model, mark_fuzzy
+ )
+ translated_count += 1
+ return translated_count, failed_count
+
+
+def backfill(
+ lang: str,
+ *,
+ batch_size: int = DEFAULT_BATCH_SIZE,
+ limit: int | None = None,
+ min_context: int = 0,
+ model: str = DEFAULT_MODEL,
+ index_path: Path = DEFAULT_INDEX,
+ dry_run: bool = False,
+ mark_fuzzy: bool = True,
+) -> None:
+ po_path = TRANSLATIONS_DIR / lang / "LC_MESSAGES" / "messages.po"
+ if not po_path.exists():
+ print(f"No .po file found for language '{lang}': {po_path}", file=sys.stderr)
+ sys.exit(1)
+ if not index_path.exists():
+ print(
+ f"Translation index not found at {index_path}.\n"
+ "Run: python scripts/translations/build_translation_index.py",
+ file=sys.stderr,
+ )
+ sys.exit(1)
+
+ print("Loading translation index …", file=sys.stderr)
+ with open(index_path, encoding="utf-8") as f:
+ index: dict[str, Any] = json.load(f)
+
+ print(f"Loading {po_path} …", file=sys.stderr)
+ cat = polib.pofile(str(po_path))
+
+ missing: list[polib.POEntry] = [e for e in cat if e.msgid and _is_missing(e)]
+ print(f"Found {len(missing)} untranslated entries for '{lang}'.", file=sys.stderr)
+
+ if min_context > 0:
+ before = len(missing)
+ missing = [
+ e
+ for e in missing
+ if _context_count(
+ {
+ "index_key": (
+ _plural_key(e.msgid, e.msgid_plural)
+ if e.msgid_plural
+ else e.msgid
+ )
+ },
+ index,
+ lang,
+ )
+ >= min_context
+ ]
+ skipped = before - len(missing)
+ print(
+ f"Skipping {skipped} entries with fewer than {min_context} reference "
+ f"translation(s) (use --min-context 0 to include them).",
+ file=sys.stderr,
+ )
+
+ if limit is not None:
+ missing = missing[:limit]
+ print(f"Limiting to {limit} entries.", file=sys.stderr)
+
+ if not missing:
+ print("Nothing to do.", file=sys.stderr)
+ return
+
+ translated_count, failed_count = _process_batches(
+ missing, index, lang, batch_size, model, dry_run, mark_fuzzy
+ )
+
+ if not dry_run and translated_count > 0:
+ print(f"\nSaving {po_path} …", file=sys.stderr)
+ cat.save()
+
+ print(
+ f"\nDone. Translated: {translated_count}, Failed/skipped: {failed_count}.",
+ file=sys.stderr,
+ )
+ if not dry_run and translated_count > 0:
+ print(
+ f"Translations written to {po_path} (marked #, fuzzy for review).",
+ file=sys.stderr,
+ )
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(
Review Comment:
Correction to my earlier reply: this function genuinely didn't have a docstring applied yet (unlike the `_lang_name`/`_plural_key` cases where the suggestion replaced the body). Added the docstring in 1ac81f7a31.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #39448:
URL: https://github.com/apache/superset/pull/39448#issuecomment-4426999165
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #39448:
URL: https://github.com/apache/superset/pull/39448#issuecomment-4274837703
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39448:
URL: https://github.com/apache/superset/pull/39448#discussion_r3101977411
##########
scripts/translations/backfill_po.py:
##########
@@ -0,0 +1,557 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Backfill missing translations in a .po file using Claude AI.
+
+For each untranslated (empty msgstr) entry in the target language, the script
+sends the English source string along with all available translations in other
+languages to Claude as context, then writes the AI-generated translation back
+into the .po file marked as #, fuzzy for human review.
+
+Usage:
+ # Build the translation index first (one-time or when .po files change)
+ python scripts/translations/build_translation_index.py
+
+ # Backfill French translations
+ python scripts/translations/backfill_po.py --lang fr
+
+ # Dry run (print what would be translated without writing)
+ python scripts/translations/backfill_po.py --lang de --dry-run
+
+ # Limit to 100 entries and use a specific model
+ python scripts/translations/backfill_po.py --lang es --limit 100 \
+ --model claude-opus-4-6
+
+Options:
+ --lang LANG ISO language code to backfill (required)
+ --batch-size N Number of strings per Claude request (default: 50)
+ --limit N Stop after translating N entries (default: unlimited)
+ --model MODEL Claude model ID (default: claude-sonnet-4-6)
+ --index PATH Path to translation_index.json (default: auto-detect)
+ --dry-run Print translations without writing to .po file
+ --fuzzy/--no-fuzzy Mark generated translations as fuzzy (default: fuzzy)
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any
+
+try:
+ import polib # type: ignore[import-untyped]
+except ImportError:
+ print("polib is required. Run: pip install polib", file=sys.stderr)
+ sys.exit(1)
+
+TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations"
+DEFAULT_INDEX = TRANSLATIONS_DIR / "translation_index.json"
+DEFAULT_MODEL = "claude-sonnet-4-6"
+DEFAULT_BATCH_SIZE = 50
+
+# Language names for the prompt, keyed by ISO code
+LANGUAGE_NAMES: dict[str, str] = {
+ "ar": "Arabic",
+ "ca": "Catalan",
+ "de": "German",
+ "es": "Spanish",
+ "fa": "Persian (Farsi)",
+ "fr": "French",
+ "it": "Italian",
+ "ja": "Japanese",
+ "ko": "Korean",
+ "mi": "Māori",
+ "nl": "Dutch",
+ "pl": "Polish",
+ "pt": "Portuguese",
+ "pt_BR": "Brazilian Portuguese",
+ "ru": "Russian",
+ "sk": "Slovak",
+ "sl": "Slovenian",
+ "tr": "Turkish",
+ "uk": "Ukrainian",
+ "zh": "Chinese (Simplified)",
+ "zh_TW": "Chinese (Traditional)",
+}
+
+
+def _lang_name(code: str) -> str:
+ return LANGUAGE_NAMES.get(code, code)
Review Comment:
**Suggestion:** Add a short docstring to this newly added helper function to satisfy the docstring requirement for new function definitions. [custom_rule]
**Severity Level:** Minor ⚠️
```suggestion
"""Return a human-readable language name for an ISO language code."""
```
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The existing code introduces a new top-level helper without a docstring, which violates the stated docstring rule for newly added functions. The improved code adds a concise, accurate docstring and preserves the function behavior unchanged.
</details>
[Fix in Cursor](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt=This%20is%20a%20comment%20left%20during%20a%20code%20review.%0A%0A%2A%2APath%3A%2A%2A%20scripts%2Ftranslations%2Fbackfill_po.py%0A%2A%2ALine%3A%2A%2A%2097%3A97%0A%2A%2AComment%3A%2A%2A%0A%09%2ACustom%20Rule%3A%20Add%20a%20short%20docstring%20to%20this%20newly%20added%20helper%20function%20to%20satisfy%20the%20docstring%20requirement%20for%20new%20function%20definitions.%0A%0AValidate%20the%20correctness%20of%20the%20flagged%20issue.%20If%20correct%2C%20How%20can%20I%20resolve%20this%3F%20If%20you%20propose%20a%20fix%2C%20implement%20it%20and%20please%20make%20it%20concise.%0A) | [Fix in VSCode Claude](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt=This%20is%20a%20comment%20left%20during%20a%20code%20review.%0A%0A%2A%2APath%3A%2A%2A%20scripts%2Ftranslations%2Fbackfill_po.py%0A%2A%2ALine%3A%2A%2A%2097%3A97%0A%2A%2AComment%3A%2A%2A%0A%09%2ACustom%20Rule%3A%20Add%20a%20short%20docstring%20to%20this%20new
ly%20added%20helper%20function%20to%20satisfy%20the%20docstring%20requirement%20for%20new%20function%20definitions.%0A%0AValidate%20the%20correctness%20of%20the%20flagged%20issue.%20If%20correct%2C%20How%20can%20I%20resolve%20this%3F%20If%20you%20propose%20a%20fix%2C%20implement%20it%20and%20please%20make%20it%20concise.%0A)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** scripts/translations/backfill_po.py
**Line:** 97:97
**Comment:**
*Custom Rule: Add a short docstring to this newly added helper function to satisfy the docstring requirement for new function definitions.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39448&comment_hash=0590902940928d91477263c570751c3c1b208d7a886a4fcb8716cb08e205527c&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39448&comment_hash=0590902940928d91477263c570751c3c1b208d7a886a4fcb8716cb08e205527c&reaction=dislike'>👎</a>
##########
scripts/translations/backfill_po.py:
##########
@@ -0,0 +1,557 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Backfill missing translations in a .po file using Claude AI.
+
+For each untranslated (empty msgstr) entry in the target language, the script
+sends the English source string along with all available translations in other
+languages to Claude as context, then writes the AI-generated translation back
+into the .po file marked as #, fuzzy for human review.
+
+Usage:
+ # Build the translation index first (one-time or when .po files change)
+ python scripts/translations/build_translation_index.py
+
+ # Backfill French translations
+ python scripts/translations/backfill_po.py --lang fr
+
+ # Dry run (print what would be translated without writing)
+ python scripts/translations/backfill_po.py --lang de --dry-run
+
+ # Limit to 100 entries and use a specific model
+ python scripts/translations/backfill_po.py --lang es --limit 100 \
+ --model claude-opus-4-6
+
+Options:
+ --lang LANG ISO language code to backfill (required)
+ --batch-size N Number of strings per Claude request (default: 50)
+ --limit N Stop after translating N entries (default: unlimited)
+ --model MODEL Claude model ID (default: claude-sonnet-4-6)
+ --index PATH Path to translation_index.json (default: auto-detect)
+ --dry-run Print translations without writing to .po file
+ --fuzzy/--no-fuzzy Mark generated translations as fuzzy (default: fuzzy)
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any
+
+try:
+ import polib # type: ignore[import-untyped]
+except ImportError:
+ print("polib is required. Run: pip install polib", file=sys.stderr)
+ sys.exit(1)
+
+TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations"
+DEFAULT_INDEX = TRANSLATIONS_DIR / "translation_index.json"
+DEFAULT_MODEL = "claude-sonnet-4-6"
+DEFAULT_BATCH_SIZE = 50
+
+# Language names for the prompt, keyed by ISO code
+LANGUAGE_NAMES: dict[str, str] = {
+ "ar": "Arabic",
+ "ca": "Catalan",
+ "de": "German",
+ "es": "Spanish",
+ "fa": "Persian (Farsi)",
+ "fr": "French",
+ "it": "Italian",
+ "ja": "Japanese",
+ "ko": "Korean",
+ "mi": "Māori",
+ "nl": "Dutch",
+ "pl": "Polish",
+ "pt": "Portuguese",
+ "pt_BR": "Brazilian Portuguese",
+ "ru": "Russian",
+ "sk": "Slovak",
+ "sl": "Slovenian",
+ "tr": "Turkish",
+ "uk": "Ukrainian",
+ "zh": "Chinese (Simplified)",
+ "zh_TW": "Chinese (Traditional)",
+}
+
+
+def _lang_name(code: str) -> str:
+ return LANGUAGE_NAMES.get(code, code)
+
+
+def _plural_key(msgid: str, msgid_plural: str) -> str:
+ return f"{msgid}\x00{msgid_plural}"
Review Comment:
**Suggestion:** Add a docstring to this new function so all newly introduced functions include documentation. [custom_rule]
**Severity Level:** Minor ⚠️
```suggestion
"""Build the translation index key used for pluralized entries."""
```
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The function is newly added and lacks a docstring in the existing code, so the suggestion correctly targets the rule violation. The proposed change adds documentation without altering execution or references.
</details>
[Fix in Cursor](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt=This%20is%20a%20comment%20left%20during%20a%20code%20review.%0A%0A%2A%2APath%3A%2A%2A%20scripts%2Ftranslations%2Fbackfill_po.py%0A%2A%2ALine%3A%2A%2A%20101%3A101%0A%2A%2AComment%3A%2A%2A%0A%09%2ACustom%20Rule%3A%20Add%20a%20docstring%20to%20this%20new%20function%20so%20all%20newly%20introduced%20functions%20include%20documentation.%0A%0AValidate%20the%20correctness%20of%20the%20flagged%20issue.%20If%20correct%2C%20How%20can%20I%20resolve%20this%3F%20If%20you%20propose%20a%20fix%2C%20implement%20it%20and%20please%20make%20it%20concise.%0A) | [Fix in VSCode Claude](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt=This%20is%20a%20comment%20left%20during%20a%20code%20review.%0A%0A%2A%2APath%3A%2A%2A%20scripts%2Ftranslations%2Fbackfill_po.py%0A%2A%2ALine%3A%2A%2A%20101%3A101%0A%2A%2AComment%3A%2A%2A%0A%09%2ACustom%20Rule%3A%20Add%20a%20docstring%20to%20this%20new%20function%20so%20all%20newly%20introduced%2
0functions%20include%20documentation.%0A%0AValidate%20the%20correctness%20of%20the%20flagged%20issue.%20If%20correct%2C%20How%20can%20I%20resolve%20this%3F%20If%20you%20propose%20a%20fix%2C%20implement%20it%20and%20please%20make%20it%20concise.%0A)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** scripts/translations/backfill_po.py
**Line:** 101:101
**Comment:**
*Custom Rule: Add a docstring to this new function so all newly introduced functions include documentation.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39448&comment_hash=18c6e44ffa7c6109cdfa72d8414489782e43d447451f3333bb13ae8d5aee8c30&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39448&comment_hash=18c6e44ffa7c6109cdfa72d8414489782e43d447451f3333bb13ae8d5aee8c30&reaction=dislike'>👎</a>
##########
scripts/translations/backfill_po.py:
##########
@@ -0,0 +1,557 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Backfill missing translations in a .po file using Claude AI.
+
+For each untranslated (empty msgstr) entry in the target language, the script
+sends the English source string along with all available translations in other
+languages to Claude as context, then writes the AI-generated translation back
+into the .po file marked as #, fuzzy for human review.
+
+Usage:
+ # Build the translation index first (one-time or when .po files change)
+ python scripts/translations/build_translation_index.py
+
+ # Backfill French translations
+ python scripts/translations/backfill_po.py --lang fr
+
+ # Dry run (print what would be translated without writing)
+ python scripts/translations/backfill_po.py --lang de --dry-run
+
+ # Limit to 100 entries and use a specific model
+ python scripts/translations/backfill_po.py --lang es --limit 100 \
+ --model claude-opus-4-6
+
+Options:
+ --lang LANG ISO language code to backfill (required)
+ --batch-size N Number of strings per Claude request (default: 50)
+ --limit N Stop after translating N entries (default: unlimited)
+ --model MODEL Claude model ID (default: claude-sonnet-4-6)
+ --index PATH Path to translation_index.json (default: auto-detect)
+ --dry-run Print translations without writing to .po file
+ --fuzzy/--no-fuzzy Mark generated translations as fuzzy (default: fuzzy)
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any
+
+try:
+ import polib # type: ignore[import-untyped]
+except ImportError:
+ print("polib is required. Run: pip install polib", file=sys.stderr)
+ sys.exit(1)
+
+TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations"
+DEFAULT_INDEX = TRANSLATIONS_DIR / "translation_index.json"
+DEFAULT_MODEL = "claude-sonnet-4-6"
+DEFAULT_BATCH_SIZE = 50
+
+# Language names for the prompt, keyed by ISO code
+LANGUAGE_NAMES: dict[str, str] = {
+ "ar": "Arabic",
+ "ca": "Catalan",
+ "de": "German",
+ "es": "Spanish",
+ "fa": "Persian (Farsi)",
+ "fr": "French",
+ "it": "Italian",
+ "ja": "Japanese",
+ "ko": "Korean",
+ "mi": "Māori",
+ "nl": "Dutch",
+ "pl": "Polish",
+ "pt": "Portuguese",
+ "pt_BR": "Brazilian Portuguese",
+ "ru": "Russian",
+ "sk": "Slovak",
+ "sl": "Slovenian",
+ "tr": "Turkish",
+ "uk": "Ukrainian",
+ "zh": "Chinese (Simplified)",
+ "zh_TW": "Chinese (Traditional)",
+}
+
+
+def _lang_name(code: str) -> str:
+ return LANGUAGE_NAMES.get(code, code)
+
+
+def _plural_key(msgid: str, msgid_plural: str) -> str:
+ return f"{msgid}\x00{msgid_plural}"
+
+
+def _is_missing(entry: polib.POEntry) -> bool:
+ """Return True for entries that need a translation."""
+ if entry.obsolete:
+ return False
+ if entry.msgid_plural:
+ return not any(v for v in entry.msgstr_plural.values())
+ return not entry.msgstr
+
+
+def _context_langs(
+ item: dict[str, Any], index: dict[str, Any], target_lang: str
+) -> list[str]:
+ """Return sorted list of language codes that have translations for this entry."""
+ key = item["index_key"]
+ if key not in index:
+ return []
+ return sorted(
+ lang for lang, val in index[key].items() if lang != target_lang and val
+ )
+
+
+def _context_count(
+ item: dict[str, Any], index: dict[str, Any], target_lang: str
+) -> int:
+ """Return the number of other-language translations available for this entry."""
+ return len(_context_langs(item, index, target_lang))
+
+
+def _render_item(
+ i: int,
+ item: dict[str, Any],
+ index: dict[str, Any],
+ target_lang: str,
+ reference_langs_sorted: list[str],
+) -> list[str]:
+ """Render one batch entry as prompt lines."""
+ lines: list[str] = []
+ ctx = _context_count(item, index, target_lang)
+ if ctx == 0:
+ lines.append(
+ f"--- [{i}] (no reference translations — translate conservatively) ---"
+ )
+ else:
+ plural = "s" if ctx != 1 else ""
+ lines.append(f"--- [{i}] ({ctx} reference translation{plural}) ---")
+ lines.append(f"English: {json.dumps(item['msgid'], ensure_ascii=False)}")
+ if item.get("msgid_plural"):
+ plural_json = json.dumps(item["msgid_plural"], ensure_ascii=False)
+ lines.append(f"English plural: {plural_json}")
+ key = item["index_key"]
+ if key in index and reference_langs_sorted:
+ for lang in reference_langs_sorted:
+ val = index[key].get(lang)
+ if val is None:
+ continue
+ if isinstance(val, dict):
+ forms = "; ".join(
+ f"[{k}] {json.dumps(v, ensure_ascii=False)}" for k, v in val.items()
+ )
+ lines.append(f"{_lang_name(lang)}: {forms}")
+ else:
+ lines.append(
+ f"{_lang_name(lang)}: {json.dumps(val, ensure_ascii=False)}"
+ )
+ lines.append("")
+ return lines
+
+
+def build_prompt(
+ target_lang: str,
+ batch: list[dict[str, Any]],
+ index: dict[str, Any],
+) -> str:
+ """Build the Claude prompt for a batch of entries."""
+ lang_name = _lang_name(target_lang)
+
+ # Collect which other languages actually have translations for this batch
+ reference_langs: set[str] = set()
+ for item in batch:
+ key = item["index_key"]
+ if key in index:
+ reference_langs.update(
+ lang for lang, val in index[key].items() if lang != target_lang and val
+ )
+ reference_langs_sorted = sorted(reference_langs)
+
+ lines: list[str] = [
+ "You are a professional translator specializing in software UI strings.",
+ f"Translate the following English strings into {lang_name} ({target_lang}).",
+ "",
+ "Rules:",
+ "- Preserve all format placeholders exactly: %(name)s, {name}, %s, %d, etc.",
+ "- Preserve HTML tags if present.",
+ "- Keep the same tone and register as the reference translations.",
+ "- For plural forms, provide translations for all plural forms"
+ " required by the language.",
+ "- Return ONLY a JSON object mapping each numeric index (as a string)"
+ " to its translation.",
+ "- Do not add any explanation, preamble, or markdown fences.",
+ "",
+ "Important: Many strings are short fragments or single words that are"
+ " ambiguous in English (e.g. 'Scale' could mean a measurement scale,"
+ " to scale an image, or fish scales). Use the translations in other"
+ " languages as your primary signal for which meaning is intended —"
+ " they collectively disambiguate the intended sense. When no"
+ " other-language translations are available for an entry, translate"
+ " conservatively based on the most common meaning in a data"
+ " visualization UI context.",
+ "",
+ ]
+
+ if reference_langs_sorted:
+ lines.append(
+ f"Reference translations are provided per string where available "
+ f"({', '.join(_lang_name(lc) for lc in reference_langs_sorted)})."
+ )
+ lines.append("")
+
+ lines.append("Strings to translate:")
+ lines.append("")
+
+ for i, item in enumerate(batch):
+ lines.extend(_render_item(i, item, index, target_lang, reference_langs_sorted))
+
+ if batch and batch[0].get("msgid_plural"):
+ # Add guidance on plural form counts per language
+ lines.append(
+ "Note: provide ALL plural forms required by the target language "
+ "(e.g. French needs 2, Russian needs 3, Arabic needs 6)."
+ )
+ lines.append("")
+
+ lines.append(
+ 'Expected output format: {"0": "<translation>", "1": "<translation>", ...}'
+ )
+ lines.append("(keys are the numeric indices of the strings above)")
+
+ return "\n".join(lines)
+
+
+def parse_response(text: str, batch_size: int) -> dict[int, str]:
+ """Parse the JSON object from Claude's response."""
+ # Strip any accidental markdown fences
+ text = re.sub(r"^```[^\n]*\n", "", text.strip())
+ text = re.sub(r"\n```$", "", text)
+ try:
+ raw = json.loads(text)
+ return {int(k): str(v) for k, v in raw.items() if str(k).isdigit()}
+ except json.JSONDecodeError as exc:
+ raise ValueError(
+ f"Could not parse response as JSON: {exc}\n\nResponse:\n{text}"
+ ) from exc
+
+
+def translate_batch(
+ model: str,
+ target_lang: str,
+ batch: list[dict[str, Any]],
+ index: dict[str, Any],
+) -> dict[int, str]:
+ """Send a batch of strings to Claude via `claude -p`.
+
+ Returns a dict mapping batch index to translated string.
+ """
+ claude_bin = shutil.which("claude")
+ if not claude_bin:
+ raise RuntimeError(
+ "claude CLI not found. Install Claude Code or add it to PATH."
+ )
+ prompt = build_prompt(target_lang, batch, index)
+ # claude_bin is resolved via shutil.which — not user-controlled input
+ result = subprocess.run( # noqa: S603
+ [claude_bin, "--model", model, "-p", prompt],
+ capture_output=True,
+ text=True,
+ check=False,
+ )
+ if result.returncode != 0:
+ raise RuntimeError(
+ f"claude exited with code {result.returncode}:\n{result.stderr}"
+ )
+ return parse_response(result.stdout.strip(), len(batch))
+
+
+def _apply_translation(
+ entry: polib.POEntry,
+ translation: str,
+ item: dict[str, Any],
+ model: str,
+ mark_fuzzy: bool,
+) -> None:
+ """Write a translation string into a POEntry and add attribution."""
+ if entry.msgid_plural:
+ # Model may return a JSON dict of plural forms or a plain string
+ try:
+ plural_dict = json.loads(translation)
+ if isinstance(plural_dict, dict):
+ entry.msgstr_plural = {int(k): str(v) for k, v in plural_dict.items()}
+ else:
+ for k in entry.msgstr_plural:
+ entry.msgstr_plural[k] = str(plural_dict)
+ except (json.JSONDecodeError, ValueError):
+ for k in entry.msgstr_plural:
+ entry.msgstr_plural[k] = translation
+ else:
+ entry.msgstr = translation
+
+ if mark_fuzzy and "fuzzy" not in entry.flags:
+ entry.flags.append("fuzzy")
+
+ refs = item["context_langs"]
+ refs_tag = f" [refs: {', '.join(refs)}]" if refs else " [no refs]"
+ attribution = f"Machine-translated via backfill_po.py ({model}){refs_tag}"
+ if entry.tcomment:
+ if attribution not in entry.tcomment:
+ entry.tcomment = f"{entry.tcomment}\n{attribution}"
+ else:
+ entry.tcomment = attribution
+
+
+def _build_batch_items(
+ entries: list[polib.POEntry],
+ index: dict[str, Any],
+ lang: str,
+) -> list[dict[str, Any]]:
+ """Convert a list of POEntries into the dict format used by translate_batch."""
+ items: list[dict[str, Any]] = []
+ for entry in entries:
+ if entry.msgid_plural:
+ item: dict[str, Any] = {
+ "msgid": entry.msgid,
+ "msgid_plural": entry.msgid_plural,
+ "index_key": _plural_key(entry.msgid, entry.msgid_plural),
+ "is_plural": True,
+ }
+ else:
+ item = {
+ "msgid": entry.msgid,
+ "index_key": entry.msgid,
+ "is_plural": False,
+ }
+ item["context_langs"] = _context_langs(item, index, lang)
+ item["context_count"] = len(item["context_langs"])
+ items.append(item)
+ return items
+
+
+def _process_batches(
+ missing: list[polib.POEntry],
+ index: dict[str, Any],
+ lang: str,
+ batch_size: int,
+ model: str,
+ dry_run: bool,
+ mark_fuzzy: bool,
+) -> tuple[int, int]:
+ """Translate missing entries in batches. Returns (translated, failed) counts."""
+ translated_count = 0
+ failed_count = 0
+ for batch_start in range(0, len(missing), batch_size):
+ batch_entries = missing[batch_start : batch_start + batch_size]
+ batch_items = _build_batch_items(batch_entries, index, lang)
+ end = min(batch_start + batch_size, len(missing))
+ print(
+ f" Translating entries {batch_start + 1}–{end} of {len(missing)} …",
+ file=sys.stderr,
+ )
+ try:
+ translations = translate_batch(model, lang, batch_items, index)
+ except (ValueError, RuntimeError) as exc:
+ print(f" ERROR in batch starting at {batch_start}: {exc}", file=sys.stderr)
+ failed_count += len(batch_entries)
+ continue
+ for i, entry in enumerate(batch_entries):
+ translation = translations.get(i)
+ if translation is None:
+ print(
+ f" WARNING: no translation returned for index {i} "
+ f"(msgid: {entry.msgid[:60]!r})",
+ file=sys.stderr,
+ )
+ failed_count += 1
+ continue
+ if dry_run:
+ ctx = batch_items[i]["context_count"]
+ ctx_tag = f" [ctx:{ctx}]" if ctx < 3 else ""
+ print(
+ f" [{lang}]{ctx_tag} {entry.msgid[:60]!r} → {translation[:60]!r}"
+ )
+ else:
+ _apply_translation(
+ entry, translation, batch_items[i], model, mark_fuzzy
+ )
+ translated_count += 1
+ return translated_count, failed_count
+
+
+def backfill(
+ lang: str,
+ *,
+ batch_size: int = DEFAULT_BATCH_SIZE,
+ limit: int | None = None,
+ min_context: int = 0,
+ model: str = DEFAULT_MODEL,
+ index_path: Path = DEFAULT_INDEX,
+ dry_run: bool = False,
+ mark_fuzzy: bool = True,
+) -> None:
Review Comment:
**Suggestion:** Add a docstring to this new top-level workflow function to comply with the requirement that newly added functions include docstrings. [custom_rule]
**Severity Level:** Minor ⚠️
```suggestion
"""Backfill missing translations in the target language .po file."""
```
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The new workflow function has no docstring in the existing code, which matches the stated requirement for new functions. The proposed docstring is syntactically valid and does not change runtime behavior.
</details>
[Fix in Cursor](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt=This%20is%20a%20comment%20left%20during%20a%20code%20review.%0A%0A%2A%2APath%3A%2A%2A%20scripts%2Ftranslations%2Fbackfill_po.py%0A%2A%2ALine%3A%2A%2A%20411%3A411%0A%2A%2AComment%3A%2A%2A%0A%09%2ACustom%20Rule%3A%20Add%20a%20docstring%20to%20this%20new%20top-level%20workflow%20function%20to%20comply%20with%20the%20requirement%20that%20newly%20added%20functions%20include%20docstrings.%0A%0AValidate%20the%20correctness%20of%20the%20flagged%20issue.%20If%20correct%2C%20How%20can%20I%20resolve%20this%3F%20If%20you%20propose%20a%20fix%2C%20implement%20it%20and%20please%20make%20it%20concise.%0A) | [Fix in VSCode Claude](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt=This%20is%20a%20comment%20left%20during%20a%20code%20review.%0A%0A%2A%2APath%3A%2A%2A%20scripts%2Ftranslations%2Fbackfill_po.py%0A%2A%2ALine%3A%2A%2A%20411%3A411%0A%2A%2AComment%3A%2A%2A%0A%09%2ACustom%20Rule%3A%20Add%20a%20docstring%20to%20thi
s%20new%20top-level%20workflow%20function%20to%20comply%20with%20the%20requirement%20that%20newly%20added%20functions%20include%20docstrings.%0A%0AValidate%20the%20correctness%20of%20the%20flagged%20issue.%20If%20correct%2C%20How%20can%20I%20resolve%20this%3F%20If%20you%20propose%20a%20fix%2C%20implement%20it%20and%20please%20make%20it%20concise.%0A)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** scripts/translations/backfill_po.py
**Line:** 411:411
**Comment:**
*Custom Rule: Add a docstring to this new top-level workflow function to comply with the requirement that newly added functions include docstrings.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39448&comment_hash=d2032bb35602ecc0d41a802ae7426ca0b35694f7284674df69982d5f3109def6&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39448&comment_hash=d2032bb35602ecc0d41a802ae7426ca0b35694f7284674df69982d5f3109def6&reaction=dislike'>👎</a>
##########
scripts/translations/backfill_po.py:
##########
@@ -0,0 +1,557 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Backfill missing translations in a .po file using Claude AI.
+
+For each untranslated (empty msgstr) entry in the target language, the script
+sends the English source string along with all available translations in other
+languages to Claude as context, then writes the AI-generated translation back
+into the .po file marked as #, fuzzy for human review.
+
+Usage:
+ # Build the translation index first (one-time or when .po files change)
+ python scripts/translations/build_translation_index.py
+
+ # Backfill French translations
+ python scripts/translations/backfill_po.py --lang fr
+
+ # Dry run (print what would be translated without writing)
+ python scripts/translations/backfill_po.py --lang de --dry-run
+
+ # Limit to 100 entries and use a specific model
+ python scripts/translations/backfill_po.py --lang es --limit 100 \
+ --model claude-opus-4-6
+
+Options:
+ --lang LANG ISO language code to backfill (required)
+ --batch-size N Number of strings per Claude request (default: 50)
+ --limit N Stop after translating N entries (default: unlimited)
+ --model MODEL Claude model ID (default: claude-sonnet-4-6)
+ --index PATH Path to translation_index.json (default: auto-detect)
+ --dry-run Print translations without writing to .po file
+ --fuzzy/--no-fuzzy Mark generated translations as fuzzy (default: fuzzy)
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any
+
+try:
+ import polib # type: ignore[import-untyped]
+except ImportError:
+ print("polib is required. Run: pip install polib", file=sys.stderr)
+ sys.exit(1)
+
+TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations"
+DEFAULT_INDEX = TRANSLATIONS_DIR / "translation_index.json"
+DEFAULT_MODEL = "claude-sonnet-4-6"
+DEFAULT_BATCH_SIZE = 50
+
+# Language names for the prompt, keyed by ISO code
+LANGUAGE_NAMES: dict[str, str] = {
+ "ar": "Arabic",
+ "ca": "Catalan",
+ "de": "German",
+ "es": "Spanish",
+ "fa": "Persian (Farsi)",
+ "fr": "French",
+ "it": "Italian",
+ "ja": "Japanese",
+ "ko": "Korean",
+ "mi": "Māori",
+ "nl": "Dutch",
+ "pl": "Polish",
+ "pt": "Portuguese",
+ "pt_BR": "Brazilian Portuguese",
+ "ru": "Russian",
+ "sk": "Slovak",
+ "sl": "Slovenian",
+ "tr": "Turkish",
+ "uk": "Ukrainian",
+ "zh": "Chinese (Simplified)",
+ "zh_TW": "Chinese (Traditional)",
+}
+
+
+def _lang_name(code: str) -> str:
+ return LANGUAGE_NAMES.get(code, code)
+
+
+def _plural_key(msgid: str, msgid_plural: str) -> str:
+ return f"{msgid}\x00{msgid_plural}"
+
+
+def _is_missing(entry: polib.POEntry) -> bool:
+ """Return True for entries that need a translation."""
+ if entry.obsolete:
+ return False
+ if entry.msgid_plural:
+ return not any(v for v in entry.msgstr_plural.values())
+ return not entry.msgstr
+
+
+def _context_langs(
+ item: dict[str, Any], index: dict[str, Any], target_lang: str
+) -> list[str]:
+ """Return sorted list of language codes that have translations for this entry."""
+ key = item["index_key"]
+ if key not in index:
+ return []
+ return sorted(
+ lang for lang, val in index[key].items() if lang != target_lang and val
+ )
+
+
+def _context_count(
+ item: dict[str, Any], index: dict[str, Any], target_lang: str
+) -> int:
+ """Return the number of other-language translations available for this entry."""
+ return len(_context_langs(item, index, target_lang))
+
+
+def _render_item(
+ i: int,
+ item: dict[str, Any],
+ index: dict[str, Any],
+ target_lang: str,
+ reference_langs_sorted: list[str],
+) -> list[str]:
+ """Render one batch entry as prompt lines."""
+ lines: list[str] = []
+ ctx = _context_count(item, index, target_lang)
+ if ctx == 0:
+ lines.append(
+ f"--- [{i}] (no reference translations — translate conservatively) ---"
+ )
+ else:
+ plural = "s" if ctx != 1 else ""
+ lines.append(f"--- [{i}] ({ctx} reference translation{plural}) ---")
+ lines.append(f"English: {json.dumps(item['msgid'], ensure_ascii=False)}")
+ if item.get("msgid_plural"):
+ plural_json = json.dumps(item["msgid_plural"], ensure_ascii=False)
+ lines.append(f"English plural: {plural_json}")
+ key = item["index_key"]
+ if key in index and reference_langs_sorted:
+ for lang in reference_langs_sorted:
+ val = index[key].get(lang)
+ if val is None:
+ continue
+ if isinstance(val, dict):
+ forms = "; ".join(
+ f"[{k}] {json.dumps(v, ensure_ascii=False)}" for k, v in val.items()
+ )
+ lines.append(f"{_lang_name(lang)}: {forms}")
+ else:
+ lines.append(
+ f"{_lang_name(lang)}: {json.dumps(val, ensure_ascii=False)}"
+ )
+ lines.append("")
+ return lines
+
+
+def build_prompt(
+ target_lang: str,
+ batch: list[dict[str, Any]],
+ index: dict[str, Any],
+) -> str:
+ """Build the Claude prompt for a batch of entries."""
+ lang_name = _lang_name(target_lang)
+
+ # Collect which other languages actually have translations for this batch
+ reference_langs: set[str] = set()
+ for item in batch:
+ key = item["index_key"]
+ if key in index:
+ reference_langs.update(
+ lang for lang, val in index[key].items() if lang != target_lang and val
+ )
+ reference_langs_sorted = sorted(reference_langs)
+
+ lines: list[str] = [
+ "You are a professional translator specializing in software UI strings.",
+ f"Translate the following English strings into {lang_name} ({target_lang}).",
+ "",
+ "Rules:",
+ "- Preserve all format placeholders exactly: %(name)s, {name}, %s, %d, etc.",
+ "- Preserve HTML tags if present.",
+ "- Keep the same tone and register as the reference translations.",
+ "- For plural forms, provide translations for all plural forms"
+ " required by the language.",
+ "- Return ONLY a JSON object mapping each numeric index (as a string)"
+ " to its translation.",
+ "- Do not add any explanation, preamble, or markdown fences.",
+ "",
+ "Important: Many strings are short fragments or single words that are"
+ " ambiguous in English (e.g. 'Scale' could mean a measurement scale,"
+ " to scale an image, or fish scales). Use the translations in other"
+ " languages as your primary signal for which meaning is intended —"
+ " they collectively disambiguate the intended sense. When no"
+ " other-language translations are available for an entry, translate"
+ " conservatively based on the most common meaning in a data"
+ " visualization UI context.",
+ "",
+ ]
+
+ if reference_langs_sorted:
+ lines.append(
+ f"Reference translations are provided per string where available "
+ f"({', '.join(_lang_name(lc) for lc in reference_langs_sorted)})."
+ )
+ lines.append("")
+
+ lines.append("Strings to translate:")
+ lines.append("")
+
+ for i, item in enumerate(batch):
+ lines.extend(_render_item(i, item, index, target_lang, reference_langs_sorted))
+
+ if batch and batch[0].get("msgid_plural"):
+ # Add guidance on plural form counts per language
+ lines.append(
+ "Note: provide ALL plural forms required by the target language "
+ "(e.g. French needs 2, Russian needs 3, Arabic needs 6)."
+ )
+ lines.append("")
+
+ lines.append(
+ 'Expected output format: {"0": "<translation>", "1": "<translation>", ...}'
+ )
+ lines.append("(keys are the numeric indices of the strings above)")
+
+ return "\n".join(lines)
+
+
+def parse_response(text: str, batch_size: int) -> dict[int, str]:
+ """Parse the JSON object from Claude's response."""
+ # Strip any accidental markdown fences
+ text = re.sub(r"^```[^\n]*\n", "", text.strip())
+ text = re.sub(r"\n```$", "", text)
+ try:
+ raw = json.loads(text)
+ return {int(k): str(v) for k, v in raw.items() if str(k).isdigit()}
+ except json.JSONDecodeError as exc:
+ raise ValueError(
+ f"Could not parse response as JSON: {exc}\n\nResponse:\n{text}"
+ ) from exc
+
+
+def translate_batch(
+ model: str,
+ target_lang: str,
+ batch: list[dict[str, Any]],
+ index: dict[str, Any],
+) -> dict[int, str]:
+ """Send a batch of strings to Claude via `claude -p`.
+
+ Returns a dict mapping batch index to translated string.
+ """
+ claude_bin = shutil.which("claude")
+ if not claude_bin:
+ raise RuntimeError(
+ "claude CLI not found. Install Claude Code or add it to PATH."
+ )
+ prompt = build_prompt(target_lang, batch, index)
+ # claude_bin is resolved via shutil.which — not user-controlled input
+ result = subprocess.run( # noqa: S603
+ [claude_bin, "--model", model, "-p", prompt],
+ capture_output=True,
+ text=True,
+ check=False,
+ )
+ if result.returncode != 0:
+ raise RuntimeError(
+ f"claude exited with code {result.returncode}:\n{result.stderr}"
+ )
+ return parse_response(result.stdout.strip(), len(batch))
+
+
+def _apply_translation(
+ entry: polib.POEntry,
+ translation: str,
+ item: dict[str, Any],
+ model: str,
+ mark_fuzzy: bool,
+) -> None:
+ """Write a translation string into a POEntry and add attribution."""
+ if entry.msgid_plural:
+ # Model may return a JSON dict of plural forms or a plain string
+ try:
+ plural_dict = json.loads(translation)
+ if isinstance(plural_dict, dict):
+ entry.msgstr_plural = {int(k): str(v) for k, v in plural_dict.items()}
+ else:
+ for k in entry.msgstr_plural:
+ entry.msgstr_plural[k] = str(plural_dict)
+ except (json.JSONDecodeError, ValueError):
+ for k in entry.msgstr_plural:
+ entry.msgstr_plural[k] = translation
+ else:
+ entry.msgstr = translation
+
+ if mark_fuzzy and "fuzzy" not in entry.flags:
+ entry.flags.append("fuzzy")
+
+ refs = item["context_langs"]
+ refs_tag = f" [refs: {', '.join(refs)}]" if refs else " [no refs]"
+ attribution = f"Machine-translated via backfill_po.py ({model}){refs_tag}"
+ if entry.tcomment:
+ if attribution not in entry.tcomment:
+ entry.tcomment = f"{entry.tcomment}\n{attribution}"
+ else:
+ entry.tcomment = attribution
+
+
+def _build_batch_items(
+ entries: list[polib.POEntry],
+ index: dict[str, Any],
+ lang: str,
+) -> list[dict[str, Any]]:
+ """Convert a list of POEntries into the dict format used by translate_batch."""
+ items: list[dict[str, Any]] = []
+ for entry in entries:
+ if entry.msgid_plural:
+ item: dict[str, Any] = {
+ "msgid": entry.msgid,
+ "msgid_plural": entry.msgid_plural,
+ "index_key": _plural_key(entry.msgid, entry.msgid_plural),
+ "is_plural": True,
+ }
+ else:
+ item = {
+ "msgid": entry.msgid,
+ "index_key": entry.msgid,
+ "is_plural": False,
+ }
+ item["context_langs"] = _context_langs(item, index, lang)
+ item["context_count"] = len(item["context_langs"])
+ items.append(item)
+ return items
+
+
+def _process_batches(
+ missing: list[polib.POEntry],
+ index: dict[str, Any],
+ lang: str,
+ batch_size: int,
+ model: str,
+ dry_run: bool,
+ mark_fuzzy: bool,
+) -> tuple[int, int]:
+ """Translate missing entries in batches. Returns (translated, failed) counts."""
+ translated_count = 0
+ failed_count = 0
+ for batch_start in range(0, len(missing), batch_size):
+ batch_entries = missing[batch_start : batch_start + batch_size]
+ batch_items = _build_batch_items(batch_entries, index, lang)
+ end = min(batch_start + batch_size, len(missing))
+ print(
+ f" Translating entries {batch_start + 1}–{end} of {len(missing)} …",
+ file=sys.stderr,
+ )
+ try:
+ translations = translate_batch(model, lang, batch_items, index)
+ except (ValueError, RuntimeError) as exc:
+ print(f" ERROR in batch starting at {batch_start}: {exc}", file=sys.stderr)
+ failed_count += len(batch_entries)
+ continue
+ for i, entry in enumerate(batch_entries):
+ translation = translations.get(i)
+ if translation is None:
+ print(
+ f" WARNING: no translation returned for index {i} "
+ f"(msgid: {entry.msgid[:60]!r})",
+ file=sys.stderr,
+ )
+ failed_count += 1
+ continue
+ if dry_run:
+ ctx = batch_items[i]["context_count"]
+ ctx_tag = f" [ctx:{ctx}]" if ctx < 3 else ""
+ print(
+ f" [{lang}]{ctx_tag} {entry.msgid[:60]!r} → {translation[:60]!r}"
+ )
+ else:
+ _apply_translation(
+ entry, translation, batch_items[i], model, mark_fuzzy
+ )
+ translated_count += 1
+ return translated_count, failed_count
+
+
+def backfill(
+ lang: str,
+ *,
+ batch_size: int = DEFAULT_BATCH_SIZE,
+ limit: int | None = None,
+ min_context: int = 0,
+ model: str = DEFAULT_MODEL,
+ index_path: Path = DEFAULT_INDEX,
+ dry_run: bool = False,
+ mark_fuzzy: bool = True,
+) -> None:
+ po_path = TRANSLATIONS_DIR / lang / "LC_MESSAGES" / "messages.po"
+ if not po_path.exists():
+ print(f"No .po file found for language '{lang}': {po_path}", file=sys.stderr)
+ sys.exit(1)
+ if not index_path.exists():
+ print(
+ f"Translation index not found at {index_path}.\n"
+ "Run: python scripts/translations/build_translation_index.py",
+ file=sys.stderr,
+ )
+ sys.exit(1)
+
+ print("Loading translation index …", file=sys.stderr)
+ with open(index_path, encoding="utf-8") as f:
+ index: dict[str, Any] = json.load(f)
+
+ print(f"Loading {po_path} …", file=sys.stderr)
+ cat = polib.pofile(str(po_path))
+
+ missing: list[polib.POEntry] = [e for e in cat if e.msgid and _is_missing(e)]
+ print(f"Found {len(missing)} untranslated entries for '{lang}'.", file=sys.stderr)
+
+ if min_context > 0:
+ before = len(missing)
+ missing = [
+ e
+ for e in missing
+ if _context_count(
+ {
+ "index_key": (
+ _plural_key(e.msgid, e.msgid_plural)
+ if e.msgid_plural
+ else e.msgid
+ )
+ },
+ index,
+ lang,
+ )
+ >= min_context
+ ]
+ skipped = before - len(missing)
+ print(
+ f"Skipping {skipped} entries with fewer than {min_context} reference "
+ f"translation(s) (use --min-context 0 to include them).",
+ file=sys.stderr,
+ )
+
+ if limit is not None:
+ missing = missing[:limit]
+ print(f"Limiting to {limit} entries.", file=sys.stderr)
+
+ if not missing:
+ print("Nothing to do.", file=sys.stderr)
+ return
+
+ translated_count, failed_count = _process_batches(
+ missing, index, lang, batch_size, model, dry_run, mark_fuzzy
+ )
+
+ if not dry_run and translated_count > 0:
+ print(f"\nSaving {po_path} …", file=sys.stderr)
+ cat.save()
+
+ print(
+ f"\nDone. Translated: {translated_count}, Failed/skipped: {failed_count}.",
+ file=sys.stderr,
+ )
+ if not dry_run and translated_count > 0:
+ print(
+ f"Translations written to {po_path} (marked #, fuzzy for review).",
+ file=sys.stderr,
+ )
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(
Review Comment:
**Suggestion:** Add a docstring to the newly added entry-point function so it meets the new-function docstring rule. [custom_rule]
**Severity Level:** Minor ⚠️
```suggestion
"""Parse CLI arguments and run translation backfill."""
```
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The entry-point function is newly introduced and currently lacks a docstring, so the suggestion addresses a real rule violation. The added docstring is valid Python and leaves the CLI logic unchanged.
</details>
[Fix in Cursor](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt=This%20is%20a%20comment%20left%20during%20a%20code%20review.%0A%0A%2A%2APath%3A%2A%2A%20scripts%2Ftranslations%2Fbackfill_po.py%0A%2A%2ALine%3A%2A%2A%20487%3A487%0A%2A%2AComment%3A%2A%2A%0A%09%2ACustom%20Rule%3A%20Add%20a%20docstring%20to%20the%20newly%20added%20entry-point%20function%20so%20it%20meets%20the%20new-function%20docstring%20rule.%0A%0AValidate%20the%20correctness%20of%20the%20flagged%20issue.%20If%20correct%2C%20How%20can%20I%20resolve%20this%3F%20If%20you%20propose%20a%20fix%2C%20implement%20it%20and%20please%20make%20it%20concise.%0A) | [Fix in VSCode Claude](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt=This%20is%20a%20comment%20left%20during%20a%20code%20review.%0A%0A%2A%2APath%3A%2A%2A%20scripts%2Ftranslations%2Fbackfill_po.py%0A%2A%2ALine%3A%2A%2A%20487%3A487%0A%2A%2AComment%3A%2A%2A%0A%09%2ACustom%20Rule%3A%20Add%20a%20docstring%20to%20the%20newly%20added%20entry-point%20function
%20so%20it%20meets%20the%20new-function%20docstring%20rule.%0A%0AValidate%20the%20correctness%20of%20the%20flagged%20issue.%20If%20correct%2C%20How%20can%20I%20resolve%20this%3F%20If%20you%20propose%20a%20fix%2C%20implement%20it%20and%20please%20make%20it%20concise.%0A)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** scripts/translations/backfill_po.py
**Line:** 487:487
**Comment:**
*Custom Rule: Add a docstring to the newly added entry-point function so it meets the new-function docstring rule.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39448&comment_hash=21a15223b0c800455db6c7bb56a0308bfa1835dfe70692e0425762023a4713dc&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39448&comment_hash=21a15223b0c800455db6c7bb56a0308bfa1835dfe70692e0425762023a4713dc&reaction=dislike'>👎</a>
##########
scripts/translations/build_translation_index.py:
##########
@@ -0,0 +1,151 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Build a cross-language translation index from all .po files.
+
+Outputs a JSON file structured as:
+ {
+ "<msgid>": {
+ "<lang>": "<translated string or null>",
+ ...
+ },
+ ...
+ }
+
+For plural entries the key is "<msgid>\x00<msgid_plural>" and the value
+is a dict mapping lang -> {0: "...", 1: "..."} (or null if untranslated).
+
+Usage:
+ python scripts/translations/build_translation_index.py
+ python scripts/translations/build_translation_index.py \
+ --translations-dir superset/translations \
+ --output /tmp/translation_index.json
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from pathlib import Path
+from typing import Any
+
+try:
+ import polib # type: ignore[import-untyped]
+except ImportError:
+ print("polib is required. Install with: pip install polib", file=sys.stderr)
+ sys.exit(1)
+
+TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations"
+DEFAULT_OUTPUT = (
+ Path(__file__).parent.parent.parent
+ / "superset"
+ / "translations"
+ / "translation_index.json"
+)
+
+
+def _is_translated(entry: polib.POEntry) -> bool:
+ """Return True if the entry has a non-empty, non-fuzzy translation."""
+ if "fuzzy" in entry.flags:
+ return False
+ if entry.msgid_plural:
+ return any(v for v in entry.msgstr_plural.values())
+ return bool(entry.msgstr)
+
+
+def _plural_key(entry: polib.POEntry) -> str:
+ return f"{entry.msgid}\x00{entry.msgid_plural}"
+
+
+def build_index(translations_dir: Path) -> dict[str, Any]:
+ """Read all .po files and build a combined translation index."""
+ index: dict[str, dict[str, Any]] = {}
+
+ langs = sorted(
+ d
+ for d in os.listdir(translations_dir)
+ if (translations_dir / d / "LC_MESSAGES" / "messages.po").exists()
+ and d != "en" # en has empty msgstr by convention (source = target)
+ )
+
+ for lang in langs:
+ po_path = translations_dir / lang / "LC_MESSAGES" / "messages.po"
+ cat = polib.pofile(str(po_path))
+ for entry in cat:
+ if not entry.msgid:
+ continue # skip header entry
+
+ if entry.msgid_plural:
+ key = _plural_key(entry)
+ if key not in index:
+ index[key] = {}
+ translated = (
+ dict(entry.msgstr_plural)
+ if any(v for v in entry.msgstr_plural.values())
+ else None
+ )
+ index[key][lang] = translated
+ else:
+ key = entry.msgid
+ if key not in index:
+ index[key] = {}
+ index[key][lang] = entry.msgstr or None
+
+ # Ensure every entry has a slot for every language (null if missing)
+ for key in index:
+ for lang in langs:
+ index[key].setdefault(lang, None)
+
+ return index
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(
Review Comment:
**Suggestion:** Add a docstring to the newly added entrypoint function to satisfy the rule requiring docstrings on new functions. [custom_rule]
**Severity Level:** Minor ⚠️
```suggestion
"""Parse arguments, build the translation index, and write it to disk."""
```
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The existing entrypoint function lacks a docstring, so it also violates the documentation rule for new functions.
The proposed docstring is valid Python syntax and does not affect runtime behavior.
</details>
[Fix in Cursor](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt=This%20is%20a%20comment%20left%20during%20a%20code%20review.%0A%0A%2A%2APath%3A%2A%2A%20scripts%2Ftranslations%2Fbuild_translation_index.py%0A%2A%2ALine%3A%2A%2A%20118%3A118%0A%2A%2AComment%3A%2A%2A%0A%09%2ACustom%20Rule%3A%20Add%20a%20docstring%20to%20the%20newly%20added%20entrypoint%20function%20to%20satisfy%20the%20rule%20requiring%20docstrings%20on%20new%20functions.%0A%0AValidate%20the%20correctness%20of%20the%20flagged%20issue.%20If%20correct%2C%20How%20can%20I%20resolve%20this%3F%20If%20you%20propose%20a%20fix%2C%20implement%20it%20and%20please%20make%20it%20concise.%0A) | [Fix in VSCode Claude](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt=This%20is%20a%20comment%20left%20during%20a%20code%20review.%0A%0A%2A%2APath%3A%2A%2A%20scripts%2Ftranslations%2Fbuild_translation_index.py%0A%2A%2ALine%3A%2A%2A%20118%3A118%0A%2A%2AComment%3A%2A%2A%0A%09%2ACustom%20Rule%3A%20Add%20a%20docstring%20to%20the
%20newly%20added%20entrypoint%20function%20to%20satisfy%20the%20rule%20requiring%20docstrings%20on%20new%20functions.%0A%0AValidate%20the%20correctness%20of%20the%20flagged%20issue.%20If%20correct%2C%20How%20can%20I%20resolve%20this%3F%20If%20you%20propose%20a%20fix%2C%20implement%20it%20and%20please%20make%20it%20concise.%0A)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** scripts/translations/build_translation_index.py
**Line:** 118:118
**Comment:**
*Custom Rule: Add a docstring to the newly added entrypoint function to satisfy the rule requiring docstrings on new functions.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39448&comment_hash=dac92a8256d6e29c178622f653e53ce629e04345498beb69d2a238b85658b51a&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39448&comment_hash=dac92a8256d6e29c178622f653e53ce629e04345498beb69d2a238b85658b51a&reaction=dislike'>👎</a>
##########
scripts/translations/build_translation_index.py:
##########
@@ -0,0 +1,151 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Build a cross-language translation index from all .po files.
+
+Outputs a JSON file structured as:
+ {
+ "<msgid>": {
+ "<lang>": "<translated string or null>",
+ ...
+ },
+ ...
+ }
+
+For plural entries the key is "<msgid>\x00<msgid_plural>" and the value
+is a dict mapping lang -> {0: "...", 1: "..."} (or null if untranslated).
+
+Usage:
+ python scripts/translations/build_translation_index.py
+ python scripts/translations/build_translation_index.py \
+ --translations-dir superset/translations \
+ --output /tmp/translation_index.json
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from pathlib import Path
+from typing import Any
+
+try:
+ import polib # type: ignore[import-untyped]
+except ImportError:
+ print("polib is required. Install with: pip install polib", file=sys.stderr)
+ sys.exit(1)
+
+TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations"
+DEFAULT_OUTPUT = (
+ Path(__file__).parent.parent.parent
+ / "superset"
+ / "translations"
+ / "translation_index.json"
+)
+
+
+def _is_translated(entry: polib.POEntry) -> bool:
+ """Return True if the entry has a non-empty, non-fuzzy translation."""
+ if "fuzzy" in entry.flags:
+ return False
+ if entry.msgid_plural:
+ return any(v for v in entry.msgstr_plural.values())
+ return bool(entry.msgstr)
+
+
+def _plural_key(entry: polib.POEntry) -> str:
+ return f"{entry.msgid}\x00{entry.msgid_plural}"
Review Comment:
**Suggestion:** Add a docstring to the newly introduced helper function so it complies with the requirement that new functions be documented. [custom_rule]
**Severity Level:** Minor ⚠️
```suggestion
"""Build the combined key used for plural translation entries."""
```
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The existing helper function has no docstring, so it violates the stated requirement that new functions be documented.
The improved_code adds a concise docstring without changing behavior and is syntactically valid.
</details>
[Fix in Cursor](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt=This%20is%20a%20comment%20left%20during%20a%20code%20review.%0A%0A%2A%2APath%3A%2A%2A%20scripts%2Ftranslations%2Fbuild_translation_index.py%0A%2A%2ALine%3A%2A%2A%2072%3A72%0A%2A%2AComment%3A%2A%2A%0A%09%2ACustom%20Rule%3A%20Add%20a%20docstring%20to%20the%20newly%20introduced%20helper%20function%20so%20it%20complies%20with%20the%20requirement%20that%20new%20functions%20be%20documented.%0A%0AValidate%20the%20correctness%20of%20the%20flagged%20issue.%20If%20correct%2C%20How%20can%20I%20resolve%20this%3F%20If%20you%20propose%20a%20fix%2C%20implement%20it%20and%20please%20make%20it%20concise.%0A) | [Fix in VSCode Claude](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt=This%20is%20a%20comment%20left%20during%20a%20code%20review.%0A%0A%2A%2APath%3A%2A%2A%20scripts%2Ftranslations%2Fbuild_translation_index.py%0A%2A%2ALine%3A%2A%2A%2072%3A72%0A%2A%2AComment%3A%2A%2A%0A%09%2ACustom%20Rule%3A%20Add%20a%20docstrin
g%20to%20the%20newly%20introduced%20helper%20function%20so%20it%20complies%20with%20the%20requirement%20that%20new%20functions%20be%20documented.%0A%0AValidate%20the%20correctness%20of%20the%20flagged%20issue.%20If%20correct%2C%20How%20can%20I%20resolve%20this%3F%20If%20you%20propose%20a%20fix%2C%20implement%20it%20and%20please%20make%20it%20concise.%0A)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** scripts/translations/build_translation_index.py
**Line:** 72:72
**Comment:**
*Custom Rule: Add a docstring to the newly introduced helper function so it complies with the requirement that new functions be documented.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39448&comment_hash=2f54a83cdd6626d9dee5d8c8a936e6d70650453938bef2c6048f52b9c38e4fa3&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39448&comment_hash=2f54a83cdd6626d9dee5d8c8a936e6d70650453938bef2c6048f52b9c38e4fa3&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "netlify[bot] (via GitHub)" <gi...@apache.org>.
netlify[bot] commented on PR #39448:
URL: https://github.com/apache/superset/pull/39448#issuecomment-4269897799
### <span aria-hidden="true">✅</span> Deploy Preview for *superset-docs-preview* ready!
| Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | 0a91bce9ea88b0a13bc3a23c53ec427b74d24fe5 |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/projects/superset-docs-preview/deploys/69e26839ea3dea00083e0ab8 |
|<span aria-hidden="true">😎</span> Deploy Preview | [https://deploy-preview-39448--superset-docs-preview.netlify.app](https://deploy-preview-39448--superset-docs-preview.netlify.app) |
|<span aria-hidden="true">📱</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br /><br /><br />_Use your smartphone camera to open QR code link._</details> |
---
<!-- [superset-docs-preview Preview](https://deploy-preview-39448--superset-docs-preview.netlify.app) -->
_To edit notification comments on pull requests, go to your [Netlify project configuration](https://app.netlify.com/projects/superset-docs-preview/configuration/notifications#deploy-notifications)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #39448:
URL: https://github.com/apache/superset/pull/39448#issuecomment-4502609965
Thanks @aminghadersohi — addressed all five in 8e69649076:
1. **Plural-note batch bug** (the real one) — `build_prompt` now gates on `any(item.get("msgid_plural") for item in batch)` instead of `batch[0]`, with a regression test (`test_build_prompt_includes_plural_note_when_plural_is_not_first`) that builds a singular-first / plural-second batch and asserts the guidance lands in the prompt.
2. **M1 `--no-fuzzy` help text** — strengthened to spell out the consequence (fuzzy entries are filtered from `.mo`; removing the flag ships AI-generated translations to end users without human review).
3. **M2 `--lang` path-traversal guard** — added `re.fullmatch(r'[a-z]{2,3}(_[A-Z]{2})?', lang)` validation before constructing the `.po` filesystem path; rejects values like `../../tmp/x` with a clear error.
4. **`--min-context` missing from module-docstring Options table** — added it alongside the other flags.
5. **Misleading test comment** — rewrote the empty-list comment to describe the actual control flow (`"[]"` parses as an empty list → falsy → scalar broadcast), not JSONDecodeError.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39448:
URL: https://github.com/apache/superset/pull/39448#discussion_r3101996757
##########
scripts/translations/build_translation_index.py:
##########
@@ -0,0 +1,151 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Build a cross-language translation index from all .po files.
+
+Outputs a JSON file structured as:
+ {
+ "<msgid>": {
+ "<lang>": "<translated string or null>",
+ ...
+ },
+ ...
+ }
+
+For plural entries the key is "<msgid>\x00<msgid_plural>" and the value
+is a dict mapping lang -> {0: "...", 1: "..."} (or null if untranslated).
+
+Usage:
+ python scripts/translations/build_translation_index.py
+ python scripts/translations/build_translation_index.py \
+ --translations-dir superset/translations \
+ --output /tmp/translation_index.json
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from pathlib import Path
+from typing import Any
+
+try:
+ import polib # type: ignore[import-untyped]
+except ImportError:
+ print("polib is required. Install with: pip install polib", file=sys.stderr)
+ sys.exit(1)
+
+TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations"
+DEFAULT_OUTPUT = (
+ Path(__file__).parent.parent.parent
+ / "superset"
+ / "translations"
+ / "translation_index.json"
+)
+
+
+def _is_translated(entry: polib.POEntry) -> bool:
+ """Return True if the entry has a non-empty, non-fuzzy translation."""
+ if "fuzzy" in entry.flags:
+ return False
+ if entry.msgid_plural:
+ return any(v for v in entry.msgstr_plural.values())
+ return bool(entry.msgstr)
+
+
+def _plural_key(entry: polib.POEntry) -> str:
+ return f"{entry.msgid}\x00{entry.msgid_plural}"
+
+
+def build_index(translations_dir: Path) -> dict[str, Any]:
+ """Read all .po files and build a combined translation index."""
+ index: dict[str, dict[str, Any]] = {}
+
+ langs = sorted(
+ d
+ for d in os.listdir(translations_dir)
+ if (translations_dir / d / "LC_MESSAGES" / "messages.po").exists()
+ and d != "en" # en has empty msgstr by convention (source = target)
+ )
+
+ for lang in langs:
+ po_path = translations_dir / lang / "LC_MESSAGES" / "messages.po"
+ cat = polib.pofile(str(po_path))
+ for entry in cat:
+ if not entry.msgid:
+ continue # skip header entry
+
+ if entry.msgid_plural:
+ key = _plural_key(entry)
+ if key not in index:
+ index[key] = {}
+ translated = (
+ dict(entry.msgstr_plural)
+ if any(v for v in entry.msgstr_plural.values())
+ else None
+ )
+ index[key][lang] = translated
+ else:
+ key = entry.msgid
+ if key not in index:
+ index[key] = {}
+ index[key][lang] = entry.msgstr or None
Review Comment:
**Suggestion:** The index currently records fuzzy translations as reference context, even though fuzzy entries are not finalized translations. This can feed AI-generated draft text back into future runs as if it were trusted context. Reuse `_is_translated` when writing index values so fuzzy entries become `null`. [logic error]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
- ⚠️ AI backfill prompt uses unreviewed machine translations as context.
- ⚠️ Min-context filtering counts fuzzy drafts as confirmed translations.
- ⚠️ Increases risk of propagating earlier AI translation mistakes.
```
</details>
```suggestion
index[key][lang] = (
dict(entry.msgstr_plural) if _is_translated(entry) else None
)
else:
key = entry.msgid
if key not in index:
index[key] = {}
index[key][lang] = entry.msgstr if _is_translated(entry) else None
```
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Run `python scripts/translations/backfill_po.py --lang es` which executes `backfill()`
in `scripts/translations/backfill_po.py:108-175`. This loads the Spanish `.po` file at
`backfill_po.py:34-52` and fills missing entries via `_apply_translation()` at
`backfill_po.py:29-63`, writing `entry.msgstr` and appending `"fuzzy"` to `entry.flags`
when `mark_fuzzy` is True (`backfill_po.py:52-53`).
2. After this run, open `superset/translations/es/LC_MESSAGES/messages.po` and observe
that new AI-generated translations have non-empty `msgstr` values and the `#, fuzzy` flag,
confirming there are fuzzy-but-filled entries which should be treated as drafts.
3. Next, rebuild the index with `python scripts/translations/build_translation_index.py`
as documented in `docs/developer_docs/contributing/howtos.md:21-24`, which calls `main()`
in `scripts/translations/build_translation_index.py:117-151` and `build_index()` at
`build_translation_index.py:75-115`. For each fuzzy plural entry, the current code at
`build_translation_index.py:93-102` sets `translated = dict(entry.msgstr_plural) if any(v
for v in entry.msgstr_plural.values()) else None`, and for non-plural entries at
`build_translation_index.py:103-107` it sets `index[key][lang] = entry.msgstr or None`,
ignoring the `fuzzy` flag and not using `_is_translated()` defined at
`build_translation_index.py:62-68`.
4. Run `python scripts/translations/backfill_po.py --lang fr --min-context 2` (Step 2/3 in
`howtos.md:31-43`), which loads `translation_index.json` in `backfill()` at
`backfill_po.py:46-48`. When computing context, `_context_langs()` at
`backfill_po.py:113-122` and `_context_count()` at `backfill_po.py:125-129` treat any
truthy `index[key][lang]` as a valid reference translation, so the fuzzy AI-generated
Spanish entries (stored as non-null strings or dicts by `build_index()`) are counted as
reference translations, feeding unreviewed machine output back as trusted context and into
the `--min-context` filtering.
```
</details>
[Fix in Cursor](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt=This%20is%20a%20comment%20left%20during%20a%20code%20review.%0A%0A%2A%2APath%3A%2A%2A%20scripts%2Ftranslations%2Fbuild_translation_index.py%0A%2A%2ALine%3A%2A%2A%2097%3A107%0A%2A%2AComment%3A%2A%2A%0A%09%2ALogic%20Error%3A%20The%20index%20currently%20records%20fuzzy%20translations%20as%20reference%20context%2C%20even%20though%20fuzzy%20entries%20are%20not%20finalized%20translations.%20This%20can%20feed%20AI-generated%20draft%20text%20back%20into%20future%20runs%20as%20if%20it%20were%20trusted%20context.%20Reuse%20%60_is_translated%60%20when%20writing%20index%20values%20so%20fuzzy%20entries%20become%20%60null%60.%0A%0AValidate%20the%20correctness%20of%20the%20flagged%20issue.%20If%20correct%2C%20How%20can%20I%20resolve%20this%3F%20If%20you%20propose%20a%20fix%2C%20implement%20it%20and%20please%20make%20it%20concise.%0A) | [Fix in VSCode Claude](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt=This%20is%
20a%20comment%20left%20during%20a%20code%20review.%0A%0A%2A%2APath%3A%2A%2A%20scripts%2Ftranslations%2Fbuild_translation_index.py%0A%2A%2ALine%3A%2A%2A%2097%3A107%0A%2A%2AComment%3A%2A%2A%0A%09%2ALogic%20Error%3A%20The%20index%20currently%20records%20fuzzy%20translations%20as%20reference%20context%2C%20even%20though%20fuzzy%20entries%20are%20not%20finalized%20translations.%20This%20can%20feed%20AI-generated%20draft%20text%20back%20into%20future%20runs%20as%20if%20it%20were%20trusted%20context.%20Reuse%20%60_is_translated%60%20when%20writing%20index%20values%20so%20fuzzy%20entries%20become%20%60null%60.%0A%0AValidate%20the%20correctness%20of%20the%20flagged%20issue.%20If%20correct%2C%20How%20can%20I%20resolve%20this%3F%20If%20you%20propose%20a%20fix%2C%20implement%20it%20and%20please%20make%20it%20concise.%0A)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** scripts/translations/build_translation_index.py
**Line:** 97:107
**Comment:**
*Logic Error: The index currently records fuzzy translations as reference context, even though fuzzy entries are not finalized translations. This can feed AI-generated draft text back into future runs as if it were trusted context. Reuse `_is_translated` when writing index values so fuzzy entries become `null`.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39448&comment_hash=729170fc82336e6b05244abd1a0040073ba5f54515245b23387e3c1148f9093f&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39448&comment_hash=729170fc82336e6b05244abd1a0040073ba5f54515245b23387e3c1148f9093f&reaction=dislike'>👎</a>
##########
scripts/translations/backfill_po.py:
##########
@@ -0,0 +1,557 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Backfill missing translations in a .po file using Claude AI.
+
+For each untranslated (empty msgstr) entry in the target language, the script
+sends the English source string along with all available translations in other
+languages to Claude as context, then writes the AI-generated translation back
+into the .po file marked as #, fuzzy for human review.
+
+Usage:
+ # Build the translation index first (one-time or when .po files change)
+ python scripts/translations/build_translation_index.py
+
+ # Backfill French translations
+ python scripts/translations/backfill_po.py --lang fr
+
+ # Dry run (print what would be translated without writing)
+ python scripts/translations/backfill_po.py --lang de --dry-run
+
+ # Limit to 100 entries and use a specific model
+ python scripts/translations/backfill_po.py --lang es --limit 100 \
+ --model claude-opus-4-6
+
+Options:
+ --lang LANG ISO language code to backfill (required)
+ --batch-size N Number of strings per Claude request (default: 50)
+ --limit N Stop after translating N entries (default: unlimited)
+ --model MODEL Claude model ID (default: claude-sonnet-4-6)
+ --index PATH Path to translation_index.json (default: auto-detect)
+ --dry-run Print translations without writing to .po file
+ --fuzzy/--no-fuzzy Mark generated translations as fuzzy (default: fuzzy)
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any
+
+try:
+ import polib # type: ignore[import-untyped]
+except ImportError:
+ print("polib is required. Run: pip install polib", file=sys.stderr)
+ sys.exit(1)
+
+TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations"
+DEFAULT_INDEX = TRANSLATIONS_DIR / "translation_index.json"
+DEFAULT_MODEL = "claude-sonnet-4-6"
+DEFAULT_BATCH_SIZE = 50
+
+# Language names for the prompt, keyed by ISO code
+LANGUAGE_NAMES: dict[str, str] = {
+ "ar": "Arabic",
+ "ca": "Catalan",
+ "de": "German",
+ "es": "Spanish",
+ "fa": "Persian (Farsi)",
+ "fr": "French",
+ "it": "Italian",
+ "ja": "Japanese",
+ "ko": "Korean",
+ "mi": "Māori",
+ "nl": "Dutch",
+ "pl": "Polish",
+ "pt": "Portuguese",
+ "pt_BR": "Brazilian Portuguese",
+ "ru": "Russian",
+ "sk": "Slovak",
+ "sl": "Slovenian",
+ "tr": "Turkish",
+ "uk": "Ukrainian",
+ "zh": "Chinese (Simplified)",
+ "zh_TW": "Chinese (Traditional)",
+}
+
+
+def _lang_name(code: str) -> str:
+ return LANGUAGE_NAMES.get(code, code)
+
+
+def _plural_key(msgid: str, msgid_plural: str) -> str:
+ return f"{msgid}\x00{msgid_plural}"
+
+
+def _is_missing(entry: polib.POEntry) -> bool:
+ """Return True for entries that need a translation."""
+ if entry.obsolete:
+ return False
+ if entry.msgid_plural:
+ return not any(v for v in entry.msgstr_plural.values())
+ return not entry.msgstr
+
+
+def _context_langs(
+ item: dict[str, Any], index: dict[str, Any], target_lang: str
+) -> list[str]:
+ """Return sorted list of language codes that have translations for this entry."""
+ key = item["index_key"]
+ if key not in index:
+ return []
+ return sorted(
+ lang for lang, val in index[key].items() if lang != target_lang and val
+ )
+
+
+def _context_count(
+ item: dict[str, Any], index: dict[str, Any], target_lang: str
+) -> int:
+ """Return the number of other-language translations available for this entry."""
+ return len(_context_langs(item, index, target_lang))
+
+
+def _render_item(
+ i: int,
+ item: dict[str, Any],
+ index: dict[str, Any],
+ target_lang: str,
+ reference_langs_sorted: list[str],
+) -> list[str]:
+ """Render one batch entry as prompt lines."""
+ lines: list[str] = []
+ ctx = _context_count(item, index, target_lang)
+ if ctx == 0:
+ lines.append(
+ f"--- [{i}] (no reference translations — translate conservatively) ---"
+ )
+ else:
+ plural = "s" if ctx != 1 else ""
+ lines.append(f"--- [{i}] ({ctx} reference translation{plural}) ---")
+ lines.append(f"English: {json.dumps(item['msgid'], ensure_ascii=False)}")
+ if item.get("msgid_plural"):
+ plural_json = json.dumps(item["msgid_plural"], ensure_ascii=False)
+ lines.append(f"English plural: {plural_json}")
+ key = item["index_key"]
+ if key in index and reference_langs_sorted:
+ for lang in reference_langs_sorted:
+ val = index[key].get(lang)
+ if val is None:
+ continue
+ if isinstance(val, dict):
+ forms = "; ".join(
+ f"[{k}] {json.dumps(v, ensure_ascii=False)}" for k, v in val.items()
+ )
+ lines.append(f"{_lang_name(lang)}: {forms}")
+ else:
+ lines.append(
+ f"{_lang_name(lang)}: {json.dumps(val, ensure_ascii=False)}"
+ )
+ lines.append("")
+ return lines
+
+
+def build_prompt(
+ target_lang: str,
+ batch: list[dict[str, Any]],
+ index: dict[str, Any],
+) -> str:
+ """Build the Claude prompt for a batch of entries."""
+ lang_name = _lang_name(target_lang)
+
+ # Collect which other languages actually have translations for this batch
+ reference_langs: set[str] = set()
+ for item in batch:
+ key = item["index_key"]
+ if key in index:
+ reference_langs.update(
+ lang for lang, val in index[key].items() if lang != target_lang and val
+ )
+ reference_langs_sorted = sorted(reference_langs)
+
+ lines: list[str] = [
+ "You are a professional translator specializing in software UI strings.",
+ f"Translate the following English strings into {lang_name} ({target_lang}).",
+ "",
+ "Rules:",
+ "- Preserve all format placeholders exactly: %(name)s, {name}, %s, %d, etc.",
+ "- Preserve HTML tags if present.",
+ "- Keep the same tone and register as the reference translations.",
+ "- For plural forms, provide translations for all plural forms"
+ " required by the language.",
+ "- Return ONLY a JSON object mapping each numeric index (as a string)"
+ " to its translation.",
+ "- Do not add any explanation, preamble, or markdown fences.",
+ "",
+ "Important: Many strings are short fragments or single words that are"
+ " ambiguous in English (e.g. 'Scale' could mean a measurement scale,"
+ " to scale an image, or fish scales). Use the translations in other"
+ " languages as your primary signal for which meaning is intended —"
+ " they collectively disambiguate the intended sense. When no"
+ " other-language translations are available for an entry, translate"
+ " conservatively based on the most common meaning in a data"
+ " visualization UI context.",
+ "",
+ ]
+
+ if reference_langs_sorted:
+ lines.append(
+ f"Reference translations are provided per string where available "
+ f"({', '.join(_lang_name(lc) for lc in reference_langs_sorted)})."
+ )
+ lines.append("")
+
+ lines.append("Strings to translate:")
+ lines.append("")
+
+ for i, item in enumerate(batch):
+ lines.extend(_render_item(i, item, index, target_lang, reference_langs_sorted))
+
+ if batch and batch[0].get("msgid_plural"):
+ # Add guidance on plural form counts per language
+ lines.append(
+ "Note: provide ALL plural forms required by the target language "
+ "(e.g. French needs 2, Russian needs 3, Arabic needs 6)."
+ )
+ lines.append("")
+
+ lines.append(
+ 'Expected output format: {"0": "<translation>", "1": "<translation>", ...}'
+ )
+ lines.append("(keys are the numeric indices of the strings above)")
+
+ return "\n".join(lines)
+
+
+def parse_response(text: str, batch_size: int) -> dict[int, str]:
+ """Parse the JSON object from Claude's response."""
+ # Strip any accidental markdown fences
+ text = re.sub(r"^```[^\n]*\n", "", text.strip())
+ text = re.sub(r"\n```$", "", text)
+ try:
+ raw = json.loads(text)
+ return {int(k): str(v) for k, v in raw.items() if str(k).isdigit()}
Review Comment:
**Suggestion:** Plural translations returned as JSON objects are coerced with `str(v)`, which turns valid JSON into Python dict syntax (`'` quotes). That breaks later `json.loads` parsing in plural handling and causes corrupted plural entries. Preserve dict/list values as JSON strings instead of Python string representations. [logic error]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
- ⚠️ Plural backfill mishandles Claude dict responses for some entries.
- ⚠️ `backfill_po.py` can write corrupted plural .po strings.
```
</details>
```suggestion
return {
int(k): (
json.dumps(v, ensure_ascii=False)
if isinstance(v, (dict, list))
else str(v)
)
for k, v in raw.items()
if str(k).isdigit()
}
```
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. From the repository root, build the translation index using the documented command
`python scripts/translations/build_translation_index.py` (see
`docs/developer_docs/contributing/howtos.md:360-363`), which writes
`superset/translations/translation_index.json` required by `backfill()`
(`scripts/translations/backfill_po.py:142-167`).
2. Ensure there is at least one untranslated plural entry (`msgid_plural` with empty
`msgstr_plural` values) in a target language `.po` file such as
`superset/translations/es/LC_MESSAGES/messages.po`, so that `backfill()` will include it
in the `missing` list (`scripts/translations/backfill_po.py:172-173`).
3. Run the backfill CLI for that language, for example `python
scripts/translations/backfill_po.py --lang es --limit 1` or via npm `npm run
translations:backfill -- --lang es --limit 1` (npm script defined in
`superset-frontend/package.json:11-13), which executes `backfill()`
(`scripts/translations/backfill_po.py:142-210`) and then `_process_batches()`
(`scripts/translations/backfill_po.py:92-139`), which calls `translate_batch()`
(`scripts/translations/backfill_po.py:258-267`).
4. When Claude returns a structured JSON value (e.g. a dict mapping plural forms) for a
plural string—behavior explicitly anticipated by the comment in `_apply_translation`
(`scripts/translations/backfill_po.py:38-42` "Model may return a JSON dict of plural forms
or a plain string")—`parse_response()` at `scripts/translations/backfill_po.py:244-252`
executes `return {int(k): str(v) for k, v in raw.items() if str(k).isdigit()}` so the dict
value `v` is converted to a Python string with single quotes; later `_apply_translation()`
calls `json.loads(translation)` on this non‑JSON string
(`scripts/translations/backfill_po.py:39-40`), which raises `json.JSONDecodeError`, is
caught in the `except` block (`scripts/translations/backfill_po.py:46-48`), and the
malformed `{...}` literal is then copied verbatim into all plural forms, silently
corrupting that plural entry in the `.po` file.
```
</details>
[Fix in Cursor](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt=This%20is%20a%20comment%20left%20during%20a%20code%20review.%0A%0A%2A%2APath%3A%2A%2A%20scripts%2Ftranslations%2Fbackfill_po.py%0A%2A%2ALine%3A%2A%2A%20251%3A251%0A%2A%2AComment%3A%2A%2A%0A%09%2ALogic%20Error%3A%20Plural%20translations%20returned%20as%20JSON%20objects%20are%20coerced%20with%20%60str%28v%29%60%2C%20which%20turns%20valid%20JSON%20into%20Python%20dict%20syntax%20%28%60%27%60%20quotes%29.%20That%20breaks%20later%20%60json.loads%60%20parsing%20in%20plural%20handling%20and%20causes%20corrupted%20plural%20entries.%20Preserve%20dict%2Flist%20values%20as%20JSON%20strings%20instead%20of%20Python%20string%20representations.%0A%0AValidate%20the%20correctness%20of%20the%20flagged%20issue.%20If%20correct%2C%20How%20can%20I%20resolve%20this%3F%20If%20you%20propose%20a%20fix%2C%20implement%20it%20and%20please%20make%20it%20concise.%0A) | [Fix in VSCode Claude](https://app.codeant.ai/fix-in-ide?tool=vscode-claude
&prompt=This%20is%20a%20comment%20left%20during%20a%20code%20review.%0A%0A%2A%2APath%3A%2A%2A%20scripts%2Ftranslations%2Fbackfill_po.py%0A%2A%2ALine%3A%2A%2A%20251%3A251%0A%2A%2AComment%3A%2A%2A%0A%09%2ALogic%20Error%3A%20Plural%20translations%20returned%20as%20JSON%20objects%20are%20coerced%20with%20%60str%28v%29%60%2C%20which%20turns%20valid%20JSON%20into%20Python%20dict%20syntax%20%28%60%27%60%20quotes%29.%20That%20breaks%20later%20%60json.loads%60%20parsing%20in%20plural%20handling%20and%20causes%20corrupted%20plural%20entries.%20Preserve%20dict%2Flist%20values%20as%20JSON%20strings%20instead%20of%20Python%20string%20representations.%0A%0AValidate%20the%20correctness%20of%20the%20flagged%20issue.%20If%20correct%2C%20How%20can%20I%20resolve%20this%3F%20If%20you%20propose%20a%20fix%2C%20implement%20it%20and%20please%20make%20it%20concise.%0A)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** scripts/translations/backfill_po.py
**Line:** 251:251
**Comment:**
*Logic Error: Plural translations returned as JSON objects are coerced with `str(v)`, which turns valid JSON into Python dict syntax (`'` quotes). That breaks later `json.loads` parsing in plural handling and causes corrupted plural entries. Preserve dict/list values as JSON strings instead of Python string representations.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39448&comment_hash=cc07f71c10a642406a84190124779dfe87180e3e41dd83c982a515749e034474&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39448&comment_hash=cc07f71c10a642406a84190124779dfe87180e3e41dd83c982a515749e034474&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #39448:
URL: https://github.com/apache/superset/pull/39448#issuecomment-4269877243
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #39448:
URL: https://github.com/apache/superset/pull/39448#discussion_r3243198307
##########
scripts/translations/build_translation_index.py:
##########
Review Comment:
Did you want to add tests for this file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39448:
URL: https://github.com/apache/superset/pull/39448#discussion_r3105994776
##########
scripts/translations/backfill_po.py:
##########
@@ -0,0 +1,571 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Backfill missing translations in a .po file using Claude AI.
+
+For each untranslated (empty msgstr) entry in the target language, the script
+sends the English source string along with all available translations in other
+languages to Claude as context, then writes the AI-generated translation back
+into the .po file marked as #, fuzzy for human review.
+
+Usage:
+ # Build the translation index first (one-time or when .po files change)
+ python scripts/translations/build_translation_index.py
+
+ # Backfill French translations
+ python scripts/translations/backfill_po.py --lang fr
+
+ # Dry run (print what would be translated without writing)
+ python scripts/translations/backfill_po.py --lang de --dry-run
+
+ # Limit to 100 entries and use a specific model
+ python scripts/translations/backfill_po.py --lang es --limit 100 \
+ --model claude-opus-4-6
+
+Options:
+ --lang LANG ISO language code to backfill (required)
+ --batch-size N Number of strings per Claude request (default: 50)
+ --limit N Stop after translating N entries (default: unlimited)
+ --model MODEL Claude model ID (default: claude-sonnet-4-6)
+ --index PATH Path to translation_index.json (default: auto-detect)
+ --dry-run Print translations without writing to .po file
+ --fuzzy/--no-fuzzy Mark generated translations as fuzzy (default: fuzzy)
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any
+
+try:
+ import polib # type: ignore[import-untyped]
+except ImportError:
+ print("polib is required. Run: pip install polib", file=sys.stderr)
+ sys.exit(1)
+
+TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations"
+DEFAULT_INDEX = TRANSLATIONS_DIR / "translation_index.json"
+DEFAULT_MODEL = "claude-sonnet-4-6"
+DEFAULT_BATCH_SIZE = 50
+
+# Language names for the prompt, keyed by ISO code
+LANGUAGE_NAMES: dict[str, str] = {
+ "ar": "Arabic",
+ "ca": "Catalan",
+ "de": "German",
+ "es": "Spanish",
+ "fa": "Persian (Farsi)",
+ "fr": "French",
+ "it": "Italian",
+ "ja": "Japanese",
+ "ko": "Korean",
+ "mi": "Māori",
+ "nl": "Dutch",
+ "pl": "Polish",
+ "pt": "Portuguese",
+ "pt_BR": "Brazilian Portuguese",
+ "ru": "Russian",
+ "sk": "Slovak",
+ "sl": "Slovenian",
+ "tr": "Turkish",
+ "uk": "Ukrainian",
+ "zh": "Chinese (Simplified)",
+ "zh_TW": "Chinese (Traditional)",
+}
+
+
+def _lang_name(code: str) -> str:
+ """Return a human-readable language name for an ISO language code."""
+ return LANGUAGE_NAMES.get(code, code)
+
+
+def _plural_key(msgid: str, msgid_plural: str) -> str:
+ """Build the translation index key used for pluralized entries."""
+ return f"{msgid}\x00{msgid_plural}"
+
+
+def _is_missing(entry: polib.POEntry) -> bool:
+ """Return True for entries that need a translation."""
+ if entry.obsolete:
+ return False
+ if entry.msgid_plural:
+ return not any(v for v in entry.msgstr_plural.values())
+ return not entry.msgstr
+
+
+def _context_langs(
+ item: dict[str, Any], index: dict[str, Any], target_lang: str
+) -> list[str]:
+ """Return sorted list of language codes that have translations for this entry."""
+ key = item["index_key"]
+ if key not in index:
+ return []
+ return sorted(
+ lang for lang, val in index[key].items() if lang != target_lang and val
+ )
+
+
+def _context_count(
+ item: dict[str, Any], index: dict[str, Any], target_lang: str
+) -> int:
+ """Return the number of other-language translations available for this entry."""
+ return len(_context_langs(item, index, target_lang))
+
+
+def _render_item(
+ i: int,
+ item: dict[str, Any],
+ index: dict[str, Any],
+ target_lang: str,
+ reference_langs_sorted: list[str],
+) -> list[str]:
+ """Render one batch entry as prompt lines."""
+ lines: list[str] = []
+ ctx = _context_count(item, index, target_lang)
+ if ctx == 0:
+ lines.append(
+ f"--- [{i}] (no reference translations — translate conservatively) ---"
+ )
+ else:
+ plural = "s" if ctx != 1 else ""
+ lines.append(f"--- [{i}] ({ctx} reference translation{plural}) ---")
+ lines.append(f"English: {json.dumps(item['msgid'], ensure_ascii=False)}")
+ if item.get("msgid_plural"):
+ plural_json = json.dumps(item["msgid_plural"], ensure_ascii=False)
+ lines.append(f"English plural: {plural_json}")
+ key = item["index_key"]
+ if key in index and reference_langs_sorted:
+ for lang in reference_langs_sorted:
+ val = index[key].get(lang)
+ if val is None:
+ continue
+ if isinstance(val, dict):
+ forms = "; ".join(
+ f"[{k}] {json.dumps(v, ensure_ascii=False)}" for k, v in val.items()
+ )
+ lines.append(f"{_lang_name(lang)}: {forms}")
+ else:
+ lines.append(
+ f"{_lang_name(lang)}: {json.dumps(val, ensure_ascii=False)}"
+ )
+ lines.append("")
+ return lines
+
+
+def build_prompt(
+ target_lang: str,
+ batch: list[dict[str, Any]],
+ index: dict[str, Any],
+) -> str:
+ """Build the Claude prompt for a batch of entries."""
+ lang_name = _lang_name(target_lang)
+
+ # Collect which other languages actually have translations for this batch
+ reference_langs: set[str] = set()
+ for item in batch:
+ key = item["index_key"]
+ if key in index:
+ reference_langs.update(
+ lang for lang, val in index[key].items() if lang != target_lang and val
+ )
+ reference_langs_sorted = sorted(reference_langs)
+
+ lines: list[str] = [
+ "You are a professional translator specializing in software UI strings.",
+ f"Translate the following English strings into {lang_name} ({target_lang}).",
+ "",
+ "Rules:",
+ "- Preserve all format placeholders exactly: %(name)s, {name}, %s, %d, etc.",
+ "- Preserve HTML tags if present.",
+ "- Keep the same tone and register as the reference translations.",
+ "- For plural forms, provide translations for all plural forms"
+ " required by the language.",
+ "- Return ONLY a JSON object mapping each numeric index (as a string)"
+ " to its translation.",
+ "- Do not add any explanation, preamble, or markdown fences.",
+ "",
+ "Important: Many strings are short fragments or single words that are"
+ " ambiguous in English (e.g. 'Scale' could mean a measurement scale,"
+ " to scale an image, or fish scales). Use the translations in other"
+ " languages as your primary signal for which meaning is intended —"
+ " they collectively disambiguate the intended sense. When no"
+ " other-language translations are available for an entry, translate"
+ " conservatively based on the most common meaning in a data"
+ " visualization UI context.",
+ "",
+ ]
+
+ if reference_langs_sorted:
+ lines.append(
+ f"Reference translations are provided per string where available "
+ f"({', '.join(_lang_name(lc) for lc in reference_langs_sorted)})."
+ )
+ lines.append("")
+
+ lines.append("Strings to translate:")
+ lines.append("")
+
+ for i, item in enumerate(batch):
+ lines.extend(_render_item(i, item, index, target_lang, reference_langs_sorted))
+
+ if batch and batch[0].get("msgid_plural"):
+ # Add guidance on plural form counts per language
+ lines.append(
+ "Note: provide ALL plural forms required by the target language "
+ "(e.g. French needs 2, Russian needs 3, Arabic needs 6)."
+ )
+ lines.append("")
+
+ lines.append(
+ 'Expected output format: {"0": "<translation>", "1": "<translation>", ...}'
+ )
+ lines.append("(keys are the numeric indices of the strings above)")
+
+ return "\n".join(lines)
+
+
+def parse_response(text: str, batch_size: int) -> dict[int, str]:
+ """Parse the JSON object from Claude's response."""
+ # Strip any accidental markdown fences
+ text = re.sub(r"^```[^\n]*\n", "", text.strip())
+ text = re.sub(r"\n```$", "", text)
+ try:
+ raw = json.loads(text)
+ # Preserve dict/list values as JSON strings so plural responses
+ # (where v is a dict of plural forms) can be re-parsed downstream
+ # by _apply_translation's json.loads. str(v) on a dict produces
+ # Python repr ({'0': 'x'}) which is not valid JSON.
+ return {
Review Comment:
**Suggestion:** The response parser assumes `json.loads` always returns an object and immediately calls `.items()`. If Claude returns valid JSON of another type (like a list or string), this raises `AttributeError`, which is not caught by the current error handling and will crash the batch run. Validate the parsed type and raise `ValueError` instead so the caller can handle it gracefully. [type error]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
- ❌ Backfill CLI can crash on malformed Claude JSON type.
- ⚠️ Batch processing in `_process_batches` aborts instead of skipping batch.
- ⚠️ Contributors running `backfill_po.py` may lose progress mid-run.
- ⚠️ Error handling expects `ValueError`, not uncaught AttributeError.
```
</details>
```suggestion
if not isinstance(raw, dict):
raise ValueError(
f"Expected a JSON object mapping indices to translations, got {type(raw).__name__}."
)
```
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Open a Python shell in the repository and import the parser: `from
scripts.translations.backfill_po import parse_response` (function defined in
`scripts/translations/backfill_po.py:246-265`).
2. Call the function with a valid JSON array (non-object) to simulate an LLM response that
returns a list instead of an object: `parse_response('[\"foo\", \"bar\"]', 2)`.
3. Inside `parse_response`, `json.loads(text)` at
`scripts/translations/backfill_po.py:252` returns a Python `list`, which is bound to
`raw`.
4. The dict comprehension at `scripts/translations/backfill_po.py:257-263` executes
`raw.items()`, raising `AttributeError: 'list' object has no attribute 'items'`, which is
not caught by the `except json.JSONDecodeError` block and would similarly crash callers
like `translate_batch()` (`scripts/translations/backfill_po.py:272-299`) and
`_process_batches()` (`scripts/translations/backfill_po.py:365-390`) when the Claude CLI
returns a non-object JSON payload.
```
</details>
[Fix in Cursor](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt=This%20is%20a%20comment%20left%20during%20a%20code%20review.%0A%0A%2A%2APath%3A%2A%2A%20scripts%2Ftranslations%2Fbackfill_po.py%0A%2A%2ALine%3A%2A%2A%20257%3A257%0A%2A%2AComment%3A%2A%2A%0A%09%2AType%20Error%3A%20The%20response%20parser%20assumes%20%60json.loads%60%20always%20returns%20an%20object%20and%20immediately%20calls%20%60.items%28%29%60.%20If%20Claude%20returns%20valid%20JSON%20of%20another%20type%20%28like%20a%20list%20or%20string%29%2C%20this%20raises%20%60AttributeError%60%2C%20which%20is%20not%20caught%20by%20the%20current%20error%20handling%20and%20will%20crash%20the%20batch%20run.%20Validate%20the%20parsed%20type%20and%20raise%20%60ValueError%60%20instead%20so%20the%20caller%20can%20handle%20it%20gracefully.%0A%0AValidate%20the%20correctness%20of%20the%20flagged%20issue.%20If%20correct%2C%20How%20can%20I%20resolve%20this%3F%20If%20you%20propose%20a%20fix%2C%20implement%20it%20and%20please%20make%20
it%20concise.%0A) | [Fix in VSCode Claude](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt=This%20is%20a%20comment%20left%20during%20a%20code%20review.%0A%0A%2A%2APath%3A%2A%2A%20scripts%2Ftranslations%2Fbackfill_po.py%0A%2A%2ALine%3A%2A%2A%20257%3A257%0A%2A%2AComment%3A%2A%2A%0A%09%2AType%20Error%3A%20The%20response%20parser%20assumes%20%60json.loads%60%20always%20returns%20an%20object%20and%20immediately%20calls%20%60.items%28%29%60.%20If%20Claude%20returns%20valid%20JSON%20of%20another%20type%20%28like%20a%20list%20or%20string%29%2C%20this%20raises%20%60AttributeError%60%2C%20which%20is%20not%20caught%20by%20the%20current%20error%20handling%20and%20will%20crash%20the%20batch%20run.%20Validate%20the%20parsed%20type%20and%20raise%20%60ValueError%60%20instead%20so%20the%20caller%20can%20handle%20it%20gracefully.%0A%0AValidate%20the%20correctness%20of%20the%20flagged%20issue.%20If%20correct%2C%20How%20can%20I%20resolve%20this%3F%20If%20you%20propose%20a%20fix%2C%20impleme
nt%20it%20and%20please%20make%20it%20concise.%0A)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** scripts/translations/backfill_po.py
**Line:** 257:257
**Comment:**
*Type Error: The response parser assumes `json.loads` always returns an object and immediately calls `.items()`. If Claude returns valid JSON of another type (like a list or string), this raises `AttributeError`, which is not caught by the current error handling and will crash the batch run. Validate the parsed type and raise `ValueError` instead so the caller can handle it gracefully.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39448&comment_hash=5ef32300594291e66cbfb5010200c7cb664f8b37a37c2e19030f352f6b5a1dec&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39448&comment_hash=5ef32300594291e66cbfb5010200c7cb664f8b37a37c2e19030f352f6b5a1dec&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas merged PR #39448:
URL: https://github.com/apache/superset/pull/39448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #39448:
URL: https://github.com/apache/superset/pull/39448#issuecomment-4446321870
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #39448:
URL: https://github.com/apache/superset/pull/39448#discussion_r3105992299
##########
scripts/translations/backfill_po.py:
##########
@@ -0,0 +1,557 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Backfill missing translations in a .po file using Claude AI.
+
+For each untranslated (empty msgstr) entry in the target language, the script
+sends the English source string along with all available translations in other
+languages to Claude as context, then writes the AI-generated translation back
+into the .po file marked as #, fuzzy for human review.
+
+Usage:
+ # Build the translation index first (one-time or when .po files change)
+ python scripts/translations/build_translation_index.py
+
+ # Backfill French translations
+ python scripts/translations/backfill_po.py --lang fr
+
+ # Dry run (print what would be translated without writing)
+ python scripts/translations/backfill_po.py --lang de --dry-run
+
+ # Limit to 100 entries and use a specific model
+ python scripts/translations/backfill_po.py --lang es --limit 100 \
+ --model claude-opus-4-6
+
+Options:
+ --lang LANG ISO language code to backfill (required)
+ --batch-size N Number of strings per Claude request (default: 50)
+ --limit N Stop after translating N entries (default: unlimited)
+ --model MODEL Claude model ID (default: claude-sonnet-4-6)
+ --index PATH Path to translation_index.json (default: auto-detect)
+ --dry-run Print translations without writing to .po file
+ --fuzzy/--no-fuzzy Mark generated translations as fuzzy (default: fuzzy)
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any
+
+try:
+ import polib # type: ignore[import-untyped]
+except ImportError:
+ print("polib is required. Run: pip install polib", file=sys.stderr)
+ sys.exit(1)
+
+TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations"
+DEFAULT_INDEX = TRANSLATIONS_DIR / "translation_index.json"
+DEFAULT_MODEL = "claude-sonnet-4-6"
+DEFAULT_BATCH_SIZE = 50
+
+# Language names for the prompt, keyed by ISO code
+LANGUAGE_NAMES: dict[str, str] = {
+ "ar": "Arabic",
+ "ca": "Catalan",
+ "de": "German",
+ "es": "Spanish",
+ "fa": "Persian (Farsi)",
+ "fr": "French",
+ "it": "Italian",
+ "ja": "Japanese",
+ "ko": "Korean",
+ "mi": "Māori",
+ "nl": "Dutch",
+ "pl": "Polish",
+ "pt": "Portuguese",
+ "pt_BR": "Brazilian Portuguese",
+ "ru": "Russian",
+ "sk": "Slovak",
+ "sl": "Slovenian",
+ "tr": "Turkish",
+ "uk": "Ukrainian",
+ "zh": "Chinese (Simplified)",
+ "zh_TW": "Chinese (Traditional)",
+}
+
+
+def _lang_name(code: str) -> str:
+ return LANGUAGE_NAMES.get(code, code)
+
+
+def _plural_key(msgid: str, msgid_plural: str) -> str:
+ return f"{msgid}\x00{msgid_plural}"
+
+
+def _is_missing(entry: polib.POEntry) -> bool:
+ """Return True for entries that need a translation."""
+ if entry.obsolete:
+ return False
+ if entry.msgid_plural:
+ return not any(v for v in entry.msgstr_plural.values())
+ return not entry.msgstr
+
+
+def _context_langs(
+ item: dict[str, Any], index: dict[str, Any], target_lang: str
+) -> list[str]:
+ """Return sorted list of language codes that have translations for this entry."""
+ key = item["index_key"]
+ if key not in index:
+ return []
+ return sorted(
+ lang for lang, val in index[key].items() if lang != target_lang and val
+ )
+
+
+def _context_count(
+ item: dict[str, Any], index: dict[str, Any], target_lang: str
+) -> int:
+ """Return the number of other-language translations available for this entry."""
+ return len(_context_langs(item, index, target_lang))
+
+
+def _render_item(
+ i: int,
+ item: dict[str, Any],
+ index: dict[str, Any],
+ target_lang: str,
+ reference_langs_sorted: list[str],
+) -> list[str]:
+ """Render one batch entry as prompt lines."""
+ lines: list[str] = []
+ ctx = _context_count(item, index, target_lang)
+ if ctx == 0:
+ lines.append(
+ f"--- [{i}] (no reference translations — translate conservatively) ---"
+ )
+ else:
+ plural = "s" if ctx != 1 else ""
+ lines.append(f"--- [{i}] ({ctx} reference translation{plural}) ---")
+ lines.append(f"English: {json.dumps(item['msgid'], ensure_ascii=False)}")
+ if item.get("msgid_plural"):
+ plural_json = json.dumps(item["msgid_plural"], ensure_ascii=False)
+ lines.append(f"English plural: {plural_json}")
+ key = item["index_key"]
+ if key in index and reference_langs_sorted:
+ for lang in reference_langs_sorted:
+ val = index[key].get(lang)
+ if val is None:
+ continue
+ if isinstance(val, dict):
+ forms = "; ".join(
+ f"[{k}] {json.dumps(v, ensure_ascii=False)}" for k, v in val.items()
+ )
+ lines.append(f"{_lang_name(lang)}: {forms}")
+ else:
+ lines.append(
+ f"{_lang_name(lang)}: {json.dumps(val, ensure_ascii=False)}"
+ )
+ lines.append("")
+ return lines
+
+
+def build_prompt(
+ target_lang: str,
+ batch: list[dict[str, Any]],
+ index: dict[str, Any],
+) -> str:
+ """Build the Claude prompt for a batch of entries."""
+ lang_name = _lang_name(target_lang)
+
+ # Collect which other languages actually have translations for this batch
+ reference_langs: set[str] = set()
+ for item in batch:
+ key = item["index_key"]
+ if key in index:
+ reference_langs.update(
+ lang for lang, val in index[key].items() if lang != target_lang and val
+ )
+ reference_langs_sorted = sorted(reference_langs)
+
+ lines: list[str] = [
+ "You are a professional translator specializing in software UI strings.",
+ f"Translate the following English strings into {lang_name} ({target_lang}).",
+ "",
+ "Rules:",
+ "- Preserve all format placeholders exactly: %(name)s, {name}, %s, %d, etc.",
+ "- Preserve HTML tags if present.",
+ "- Keep the same tone and register as the reference translations.",
+ "- For plural forms, provide translations for all plural forms"
+ " required by the language.",
+ "- Return ONLY a JSON object mapping each numeric index (as a string)"
+ " to its translation.",
+ "- Do not add any explanation, preamble, or markdown fences.",
+ "",
+ "Important: Many strings are short fragments or single words that are"
+ " ambiguous in English (e.g. 'Scale' could mean a measurement scale,"
+ " to scale an image, or fish scales). Use the translations in other"
+ " languages as your primary signal for which meaning is intended —"
+ " they collectively disambiguate the intended sense. When no"
+ " other-language translations are available for an entry, translate"
+ " conservatively based on the most common meaning in a data"
+ " visualization UI context.",
+ "",
+ ]
+
+ if reference_langs_sorted:
+ lines.append(
+ f"Reference translations are provided per string where available "
+ f"({', '.join(_lang_name(lc) for lc in reference_langs_sorted)})."
+ )
+ lines.append("")
+
+ lines.append("Strings to translate:")
+ lines.append("")
+
+ for i, item in enumerate(batch):
+ lines.extend(_render_item(i, item, index, target_lang, reference_langs_sorted))
+
+ if batch and batch[0].get("msgid_plural"):
+ # Add guidance on plural form counts per language
+ lines.append(
+ "Note: provide ALL plural forms required by the target language "
+ "(e.g. French needs 2, Russian needs 3, Arabic needs 6)."
+ )
+ lines.append("")
+
+ lines.append(
+ 'Expected output format: {"0": "<translation>", "1": "<translation>", ...}'
+ )
+ lines.append("(keys are the numeric indices of the strings above)")
+
+ return "\n".join(lines)
+
+
+def parse_response(text: str, batch_size: int) -> dict[int, str]:
+ """Parse the JSON object from Claude's response."""
+ # Strip any accidental markdown fences
+ text = re.sub(r"^```[^\n]*\n", "", text.strip())
+ text = re.sub(r"\n```$", "", text)
+ try:
+ raw = json.loads(text)
+ return {int(k): str(v) for k, v in raw.items() if str(k).isdigit()}
Review Comment:
Good catch — applied in 37af05c29.
`parse_response` now serializes dict/list values with `json.dumps(..., ensure_ascii=False)` before casting, so plural responses round-trip correctly through the `json.loads(translation)` call in `_apply_translation`. Scalar values still go through `str(v)` for backward compatibility with single-string responses.
##########
scripts/translations/build_translation_index.py:
##########
@@ -0,0 +1,151 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Build a cross-language translation index from all .po files.
+
+Outputs a JSON file structured as:
+ {
+ "<msgid>": {
+ "<lang>": "<translated string or null>",
+ ...
+ },
+ ...
+ }
+
+For plural entries the key is "<msgid>\x00<msgid_plural>" and the value
+is a dict mapping lang -> {0: "...", 1: "..."} (or null if untranslated).
+
+Usage:
+ python scripts/translations/build_translation_index.py
+ python scripts/translations/build_translation_index.py \
+ --translations-dir superset/translations \
+ --output /tmp/translation_index.json
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from pathlib import Path
+from typing import Any
+
+try:
+ import polib # type: ignore[import-untyped]
+except ImportError:
+ print("polib is required. Install with: pip install polib", file=sys.stderr)
+ sys.exit(1)
+
+TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations"
+DEFAULT_OUTPUT = (
+ Path(__file__).parent.parent.parent
+ / "superset"
+ / "translations"
+ / "translation_index.json"
+)
+
+
+def _is_translated(entry: polib.POEntry) -> bool:
+ """Return True if the entry has a non-empty, non-fuzzy translation."""
+ if "fuzzy" in entry.flags:
+ return False
+ if entry.msgid_plural:
+ return any(v for v in entry.msgstr_plural.values())
+ return bool(entry.msgstr)
+
+
+def _plural_key(entry: polib.POEntry) -> str:
+ return f"{entry.msgid}\x00{entry.msgid_plural}"
+
+
+def build_index(translations_dir: Path) -> dict[str, Any]:
+ """Read all .po files and build a combined translation index."""
+ index: dict[str, dict[str, Any]] = {}
+
+ langs = sorted(
+ d
+ for d in os.listdir(translations_dir)
+ if (translations_dir / d / "LC_MESSAGES" / "messages.po").exists()
+ and d != "en" # en has empty msgstr by convention (source = target)
+ )
+
+ for lang in langs:
+ po_path = translations_dir / lang / "LC_MESSAGES" / "messages.po"
+ cat = polib.pofile(str(po_path))
+ for entry in cat:
+ if not entry.msgid:
+ continue # skip header entry
+
+ if entry.msgid_plural:
+ key = _plural_key(entry)
+ if key not in index:
+ index[key] = {}
+ translated = (
+ dict(entry.msgstr_plural)
+ if any(v for v in entry.msgstr_plural.values())
+ else None
+ )
+ index[key][lang] = translated
+ else:
+ key = entry.msgid
+ if key not in index:
+ index[key] = {}
+ index[key][lang] = entry.msgstr or None
Review Comment:
Great catch — applied in 37af05c29.
`build_index` now routes both singular and plural entries through `_is_translated`, so fuzzy entries get written as `null`. This prevents AI-generated drafts (which are always marked fuzzy on write) from feeding back into future runs as trusted context, and also keeps `--min-context N` filtering honest (fuzzy drafts no longer count toward the reference threshold).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #39448:
URL: https://github.com/apache/superset/pull/39448#discussion_r3244220979
##########
scripts/translations/build_translation_index.py:
##########
Review Comment:
Good call — added in de4c929e44.
17 tests under `tests/unit_tests/scripts/translations/build_translation_index_test.py` covering the two pure helpers (`_is_translated`, `_plural_key`) and the `build_index` workflow against a `tmp_path` fixture with three fake languages.
The tests lock in the cross-language index contract that the backfill prompt relies on:
- fuzzy entries surface as `null` (so unreviewed drafts don't feed back into the prompt as trusted context, and don't inflate `--min-context`)
- every msgid has a slot for every non-en language (`null` when missing)
- plural entries are keyed by the `msgid\x00msgid_plural` composite
- the `en` source locale and directories without `LC_MESSAGES/messages.po` are excluded
- the `.po` header entry (empty msgid) is skipped
Pairs with the existing `backfill_po_test.py` (already on the PR) which covers `parse_response` and `_apply_translation` — together they cover both scripts in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #39448:
URL: https://github.com/apache/superset/pull/39448#discussion_r3111945491
##########
scripts/translations/backfill_po.py:
##########
@@ -0,0 +1,571 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Backfill missing translations in a .po file using Claude AI.
+
+For each untranslated (empty msgstr) entry in the target language, the script
+sends the English source string along with all available translations in other
+languages to Claude as context, then writes the AI-generated translation back
+into the .po file marked as #, fuzzy for human review.
+
+Usage:
+ # Build the translation index first (one-time or when .po files change)
+ python scripts/translations/build_translation_index.py
+
+ # Backfill French translations
+ python scripts/translations/backfill_po.py --lang fr
+
+ # Dry run (print what would be translated without writing)
+ python scripts/translations/backfill_po.py --lang de --dry-run
+
+ # Limit to 100 entries and use a specific model
+ python scripts/translations/backfill_po.py --lang es --limit 100 \
+ --model claude-opus-4-6
+
+Options:
+ --lang LANG ISO language code to backfill (required)
+ --batch-size N Number of strings per Claude request (default: 50)
+ --limit N Stop after translating N entries (default: unlimited)
+ --model MODEL Claude model ID (default: claude-sonnet-4-6)
+ --index PATH Path to translation_index.json (default: auto-detect)
+ --dry-run Print translations without writing to .po file
+ --fuzzy/--no-fuzzy Mark generated translations as fuzzy (default: fuzzy)
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any
+
+try:
+ import polib # type: ignore[import-untyped]
+except ImportError:
+ print("polib is required. Run: pip install polib", file=sys.stderr)
+ sys.exit(1)
+
+TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations"
+DEFAULT_INDEX = TRANSLATIONS_DIR / "translation_index.json"
+DEFAULT_MODEL = "claude-sonnet-4-6"
+DEFAULT_BATCH_SIZE = 50
+
+# Language names for the prompt, keyed by ISO code
+LANGUAGE_NAMES: dict[str, str] = {
+ "ar": "Arabic",
+ "ca": "Catalan",
+ "de": "German",
+ "es": "Spanish",
+ "fa": "Persian (Farsi)",
+ "fr": "French",
+ "it": "Italian",
+ "ja": "Japanese",
+ "ko": "Korean",
+ "mi": "Māori",
+ "nl": "Dutch",
+ "pl": "Polish",
+ "pt": "Portuguese",
+ "pt_BR": "Brazilian Portuguese",
+ "ru": "Russian",
+ "sk": "Slovak",
+ "sl": "Slovenian",
+ "tr": "Turkish",
+ "uk": "Ukrainian",
+ "zh": "Chinese (Simplified)",
+ "zh_TW": "Chinese (Traditional)",
+}
+
+
+def _lang_name(code: str) -> str:
+ """Return a human-readable language name for an ISO language code."""
+ return LANGUAGE_NAMES.get(code, code)
+
+
+def _plural_key(msgid: str, msgid_plural: str) -> str:
+ """Build the translation index key used for pluralized entries."""
+ return f"{msgid}\x00{msgid_plural}"
+
+
+def _is_missing(entry: polib.POEntry) -> bool:
+ """Return True for entries that need a translation."""
+ if entry.obsolete:
+ return False
+ if entry.msgid_plural:
+ return not any(v for v in entry.msgstr_plural.values())
+ return not entry.msgstr
+
+
+def _context_langs(
+ item: dict[str, Any], index: dict[str, Any], target_lang: str
+) -> list[str]:
+ """Return sorted list of language codes that have translations for this entry."""
+ key = item["index_key"]
+ if key not in index:
+ return []
+ return sorted(
+ lang for lang, val in index[key].items() if lang != target_lang and val
+ )
+
+
+def _context_count(
+ item: dict[str, Any], index: dict[str, Any], target_lang: str
+) -> int:
+ """Return the number of other-language translations available for this entry."""
+ return len(_context_langs(item, index, target_lang))
+
+
+def _render_item(
+ i: int,
+ item: dict[str, Any],
+ index: dict[str, Any],
+ target_lang: str,
+ reference_langs_sorted: list[str],
+) -> list[str]:
+ """Render one batch entry as prompt lines."""
+ lines: list[str] = []
+ ctx = _context_count(item, index, target_lang)
+ if ctx == 0:
+ lines.append(
+ f"--- [{i}] (no reference translations — translate conservatively) ---"
+ )
+ else:
+ plural = "s" if ctx != 1 else ""
+ lines.append(f"--- [{i}] ({ctx} reference translation{plural}) ---")
+ lines.append(f"English: {json.dumps(item['msgid'], ensure_ascii=False)}")
+ if item.get("msgid_plural"):
+ plural_json = json.dumps(item["msgid_plural"], ensure_ascii=False)
+ lines.append(f"English plural: {plural_json}")
+ key = item["index_key"]
+ if key in index and reference_langs_sorted:
+ for lang in reference_langs_sorted:
+ val = index[key].get(lang)
+ if val is None:
+ continue
+ if isinstance(val, dict):
+ forms = "; ".join(
+ f"[{k}] {json.dumps(v, ensure_ascii=False)}" for k, v in val.items()
+ )
+ lines.append(f"{_lang_name(lang)}: {forms}")
+ else:
+ lines.append(
+ f"{_lang_name(lang)}: {json.dumps(val, ensure_ascii=False)}"
+ )
+ lines.append("")
+ return lines
+
+
+def build_prompt(
+ target_lang: str,
+ batch: list[dict[str, Any]],
+ index: dict[str, Any],
+) -> str:
+ """Build the Claude prompt for a batch of entries."""
+ lang_name = _lang_name(target_lang)
+
+ # Collect which other languages actually have translations for this batch
+ reference_langs: set[str] = set()
+ for item in batch:
+ key = item["index_key"]
+ if key in index:
+ reference_langs.update(
+ lang for lang, val in index[key].items() if lang != target_lang and val
+ )
+ reference_langs_sorted = sorted(reference_langs)
+
+ lines: list[str] = [
+ "You are a professional translator specializing in software UI strings.",
+ f"Translate the following English strings into {lang_name} ({target_lang}).",
+ "",
+ "Rules:",
+ "- Preserve all format placeholders exactly: %(name)s, {name}, %s, %d, etc.",
+ "- Preserve HTML tags if present.",
+ "- Keep the same tone and register as the reference translations.",
+ "- For plural forms, provide translations for all plural forms"
+ " required by the language.",
+ "- Return ONLY a JSON object mapping each numeric index (as a string)"
+ " to its translation.",
+ "- Do not add any explanation, preamble, or markdown fences.",
+ "",
+ "Important: Many strings are short fragments or single words that are"
+ " ambiguous in English (e.g. 'Scale' could mean a measurement scale,"
+ " to scale an image, or fish scales). Use the translations in other"
+ " languages as your primary signal for which meaning is intended —"
+ " they collectively disambiguate the intended sense. When no"
+ " other-language translations are available for an entry, translate"
+ " conservatively based on the most common meaning in a data"
+ " visualization UI context.",
+ "",
+ ]
+
+ if reference_langs_sorted:
+ lines.append(
+ f"Reference translations are provided per string where available "
+ f"({', '.join(_lang_name(lc) for lc in reference_langs_sorted)})."
+ )
+ lines.append("")
+
+ lines.append("Strings to translate:")
+ lines.append("")
+
+ for i, item in enumerate(batch):
+ lines.extend(_render_item(i, item, index, target_lang, reference_langs_sorted))
+
+ if batch and batch[0].get("msgid_plural"):
+ # Add guidance on plural form counts per language
+ lines.append(
+ "Note: provide ALL plural forms required by the target language "
+ "(e.g. French needs 2, Russian needs 3, Arabic needs 6)."
+ )
+ lines.append("")
+
+ lines.append(
+ 'Expected output format: {"0": "<translation>", "1": "<translation>", ...}'
+ )
+ lines.append("(keys are the numeric indices of the strings above)")
+
+ return "\n".join(lines)
+
+
+def parse_response(text: str, batch_size: int) -> dict[int, str]:
+ """Parse the JSON object from Claude's response."""
+ # Strip any accidental markdown fences
+ text = re.sub(r"^```[^\n]*\n", "", text.strip())
+ text = re.sub(r"\n```$", "", text)
+ try:
+ raw = json.loads(text)
+ # Preserve dict/list values as JSON strings so plural responses
+ # (where v is a dict of plural forms) can be re-parsed downstream
+ # by _apply_translation's json.loads. str(v) on a dict produces
+ # Python repr ({'0': 'x'}) which is not valid JSON.
+ return {
Review Comment:
Good catch — fixed in 73a042ed5c.
A non-dict `raw` (list/scalar/null) would indeed raise `AttributeError` from `.items()`. Since `_process_batches` only catches `(ValueError, RuntimeError)` at backfill_po.py:387, that would crash the whole run instead of being handled per-batch. The fix surfaces the type error as a `ValueError` so it's caught gracefully and the run continues with the next batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #39448:
URL: https://github.com/apache/superset/pull/39448#issuecomment-4272435593
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #39448:
URL: https://github.com/apache/superset/pull/39448#issuecomment-4276765581
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #39448:
URL: https://github.com/apache/superset/pull/39448#issuecomment-4282728171
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #39448:
URL: https://github.com/apache/superset/pull/39448#discussion_r3133190341
##########
scripts/translations/backfill_po.py:
##########
@@ -0,0 +1,571 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Backfill missing translations in a .po file using Claude AI.
+
+For each untranslated (empty msgstr) entry in the target language, the script
+sends the English source string along with all available translations in other
+languages to Claude as context, then writes the AI-generated translation back
+into the .po file marked as #, fuzzy for human review.
+
+Usage:
+ # Build the translation index first (one-time or when .po files change)
+ python scripts/translations/build_translation_index.py
+
+ # Backfill French translations
+ python scripts/translations/backfill_po.py --lang fr
+
+ # Dry run (print what would be translated without writing)
+ python scripts/translations/backfill_po.py --lang de --dry-run
+
+ # Limit to 100 entries and use a specific model
+ python scripts/translations/backfill_po.py --lang es --limit 100 \
+ --model claude-opus-4-6
+
+Options:
+ --lang LANG ISO language code to backfill (required)
+ --batch-size N Number of strings per Claude request (default: 50)
+ --limit N Stop after translating N entries (default: unlimited)
+ --model MODEL Claude model ID (default: claude-sonnet-4-6)
+ --index PATH Path to translation_index.json (default: auto-detect)
+ --dry-run Print translations without writing to .po file
+ --fuzzy/--no-fuzzy Mark generated translations as fuzzy (default: fuzzy)
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any
+
+try:
+ import polib # type: ignore[import-untyped]
+except ImportError:
+ print("polib is required. Run: pip install polib", file=sys.stderr)
+ sys.exit(1)
+
+TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations"
+DEFAULT_INDEX = TRANSLATIONS_DIR / "translation_index.json"
+DEFAULT_MODEL = "claude-sonnet-4-6"
+DEFAULT_BATCH_SIZE = 50
+
+# Language names for the prompt, keyed by ISO code
+LANGUAGE_NAMES: dict[str, str] = {
+ "ar": "Arabic",
+ "ca": "Catalan",
+ "de": "German",
+ "es": "Spanish",
+ "fa": "Persian (Farsi)",
+ "fr": "French",
+ "it": "Italian",
+ "ja": "Japanese",
+ "ko": "Korean",
+ "mi": "Māori",
+ "nl": "Dutch",
+ "pl": "Polish",
+ "pt": "Portuguese",
+ "pt_BR": "Brazilian Portuguese",
+ "ru": "Russian",
+ "sk": "Slovak",
+ "sl": "Slovenian",
+ "tr": "Turkish",
+ "uk": "Ukrainian",
+ "zh": "Chinese (Simplified)",
+ "zh_TW": "Chinese (Traditional)",
+}
+
+
+def _lang_name(code: str) -> str:
+ """Return a human-readable language name for an ISO language code."""
+ return LANGUAGE_NAMES.get(code, code)
+
+
+def _plural_key(msgid: str, msgid_plural: str) -> str:
+ """Build the translation index key used for pluralized entries."""
+ return f"{msgid}\x00{msgid_plural}"
+
+
+def _is_missing(entry: polib.POEntry) -> bool:
+ """Return True for entries that need a translation."""
+ if entry.obsolete:
+ return False
+ if entry.msgid_plural:
+ return not any(v for v in entry.msgstr_plural.values())
+ return not entry.msgstr
+
+
+def _context_langs(
+ item: dict[str, Any], index: dict[str, Any], target_lang: str
+) -> list[str]:
+ """Return sorted list of language codes that have translations for this entry."""
+ key = item["index_key"]
+ if key not in index:
+ return []
+ return sorted(
+ lang for lang, val in index[key].items() if lang != target_lang and val
+ )
+
+
+def _context_count(
+ item: dict[str, Any], index: dict[str, Any], target_lang: str
+) -> int:
+ """Return the number of other-language translations available for this entry."""
+ return len(_context_langs(item, index, target_lang))
+
+
+def _render_item(
+ i: int,
+ item: dict[str, Any],
+ index: dict[str, Any],
+ target_lang: str,
+ reference_langs_sorted: list[str],
+) -> list[str]:
+ """Render one batch entry as prompt lines."""
+ lines: list[str] = []
+ ctx = _context_count(item, index, target_lang)
+ if ctx == 0:
+ lines.append(
+ f"--- [{i}] (no reference translations — translate conservatively) ---"
+ )
+ else:
+ plural = "s" if ctx != 1 else ""
+ lines.append(f"--- [{i}] ({ctx} reference translation{plural}) ---")
+ lines.append(f"English: {json.dumps(item['msgid'], ensure_ascii=False)}")
+ if item.get("msgid_plural"):
+ plural_json = json.dumps(item["msgid_plural"], ensure_ascii=False)
+ lines.append(f"English plural: {plural_json}")
+ key = item["index_key"]
+ if key in index and reference_langs_sorted:
+ for lang in reference_langs_sorted:
+ val = index[key].get(lang)
+ if val is None:
+ continue
+ if isinstance(val, dict):
+ forms = "; ".join(
+ f"[{k}] {json.dumps(v, ensure_ascii=False)}" for k, v in val.items()
+ )
+ lines.append(f"{_lang_name(lang)}: {forms}")
+ else:
+ lines.append(
+ f"{_lang_name(lang)}: {json.dumps(val, ensure_ascii=False)}"
+ )
+ lines.append("")
+ return lines
+
+
+def build_prompt(
+ target_lang: str,
+ batch: list[dict[str, Any]],
+ index: dict[str, Any],
+) -> str:
+ """Build the Claude prompt for a batch of entries."""
+ lang_name = _lang_name(target_lang)
+
+ # Collect which other languages actually have translations for this batch
+ reference_langs: set[str] = set()
+ for item in batch:
+ key = item["index_key"]
+ if key in index:
+ reference_langs.update(
+ lang for lang, val in index[key].items() if lang != target_lang and val
+ )
+ reference_langs_sorted = sorted(reference_langs)
+
+ lines: list[str] = [
+ "You are a professional translator specializing in software UI strings.",
+ f"Translate the following English strings into {lang_name} ({target_lang}).",
+ "",
+ "Rules:",
+ "- Preserve all format placeholders exactly: %(name)s, {name}, %s, %d, etc.",
+ "- Preserve HTML tags if present.",
+ "- Keep the same tone and register as the reference translations.",
+ "- For plural forms, provide translations for all plural forms"
+ " required by the language.",
+ "- Return ONLY a JSON object mapping each numeric index (as a string)"
+ " to its translation.",
+ "- Do not add any explanation, preamble, or markdown fences.",
+ "",
+ "Important: Many strings are short fragments or single words that are"
+ " ambiguous in English (e.g. 'Scale' could mean a measurement scale,"
+ " to scale an image, or fish scales). Use the translations in other"
+ " languages as your primary signal for which meaning is intended —"
+ " they collectively disambiguate the intended sense. When no"
+ " other-language translations are available for an entry, translate"
+ " conservatively based on the most common meaning in a data"
+ " visualization UI context.",
+ "",
+ ]
+
+ if reference_langs_sorted:
+ lines.append(
+ f"Reference translations are provided per string where available "
+ f"({', '.join(_lang_name(lc) for lc in reference_langs_sorted)})."
+ )
+ lines.append("")
+
+ lines.append("Strings to translate:")
+ lines.append("")
+
+ for i, item in enumerate(batch):
+ lines.extend(_render_item(i, item, index, target_lang, reference_langs_sorted))
+
+ if batch and batch[0].get("msgid_plural"):
+ # Add guidance on plural form counts per language
+ lines.append(
+ "Note: provide ALL plural forms required by the target language "
+ "(e.g. French needs 2, Russian needs 3, Arabic needs 6)."
+ )
+ lines.append("")
+
+ lines.append(
+ 'Expected output format: {"0": "<translation>", "1": "<translation>", ...}'
+ )
+ lines.append("(keys are the numeric indices of the strings above)")
+
+ return "\n".join(lines)
+
+
+def parse_response(text: str, batch_size: int) -> dict[int, str]:
+ """Parse the JSON object from Claude's response."""
+ # Strip any accidental markdown fences
+ text = re.sub(r"^```[^\n]*\n", "", text.strip())
+ text = re.sub(r"\n```$", "", text)
+ try:
+ raw = json.loads(text)
+ # Preserve dict/list values as JSON strings so plural responses
+ # (where v is a dict of plural forms) can be re-parsed downstream
+ # by _apply_translation's json.loads. str(v) on a dict produces
+ # Python repr ({'0': 'x'}) which is not valid JSON.
+ return {
Review Comment:
Already addressed in 73a042e — `parse_response` now checks `isinstance(raw, dict)` and raises `ValueError` so `_process_batches` catches it instead of crashing on `AttributeError`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #39448:
URL: https://github.com/apache/superset/pull/39448#issuecomment-4426131825
Thanks @sadpandajoe — addressed all four in 795d6e67df.
1. **Tests added** — `tests/unit_tests/scripts/translations/backfill_po_test.py` (19 cases) covers `parse_response` (singular/plural/markdown-fence stripping/non-ASCII/non-numeric keys/list-and-scalar rejection/JSON errors) and `_apply_translation` (singular path, plural-dict, plural-scalar fallback, plural invalid-JSON fallback, fuzzy flag, attribution append/dedup, end-to-end round-trip from `parse_response` into `_apply_translation`). The script is loaded via `importlib` since it lives outside the package tree.
2. **Stdin instead of argv** — `translate_batch` now uses `subprocess.run([claude_bin, '--model', model, '-p'], input=prompt, ...)`. Removes the ARG_MAX ceiling for batches with many reference languages.
3. **Save after each batch** — `_process_batches` now calls `cat.save()` after each batch that wrote at least one translation (when not in `--dry-run`). The single end-of-run `cat.save()` in `backfill()` is removed since the per-batch save covers it. A crash mid-run now only loses the in-flight batch.
4. **Docstring fixed** — module docstring's `--fuzzy/--no-fuzzy` line now reads `--no-fuzzy Do not mark generated translations as fuzzy (default: mark fuzzy)` to match what argparse actually registers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #39448:
URL: https://github.com/apache/superset/pull/39448#issuecomment-4458817399
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #39448:
URL: https://github.com/apache/superset/pull/39448#issuecomment-4269898234
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/39448?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 54.60%. Comparing base ([`4bdc8d4`](https://app.codecov.io/gh/apache/superset/commit/4bdc8d4c68102eccc1c3ffeeaa442d896b1f7a31?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`0a91bce`](https://app.codecov.io/gh/apache/superset/commit/0a91bce9ea88b0a13bc3a23c53ec427b74d24fe5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #39448 +/- ##
==========================================
- Coverage 64.49% 54.60% -9.90%
==========================================
Files 2557 2557
Lines 133097 133097
Branches 30910 30910
==========================================
- Hits 85846 72677 -13169
- Misses 45761 60085 +14324
+ Partials 1490 335 -1155
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/39448/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/39448/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `39.87% <ø> (ø)` | |
| [mysql](https://app.codecov.io/gh/apache/superset/pull/39448/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [postgres](https://app.codecov.io/gh/apache/superset/pull/39448/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [presto](https://app.codecov.io/gh/apache/superset/pull/39448/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [python](https://app.codecov.io/gh/apache/superset/pull/39448/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `39.87% <ø> (-22.29%)` | :arrow_down: |
| [sqlite](https://app.codecov.io/gh/apache/superset/pull/39448/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unit](https://app.codecov.io/gh/apache/superset/pull/39448/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/superset/pull/39448?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #39448:
URL: https://github.com/apache/superset/pull/39448#discussion_r3105992047
##########
scripts/translations/backfill_po.py:
##########
@@ -0,0 +1,557 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Backfill missing translations in a .po file using Claude AI.
+
+For each untranslated (empty msgstr) entry in the target language, the script
+sends the English source string along with all available translations in other
+languages to Claude as context, then writes the AI-generated translation back
+into the .po file marked as #, fuzzy for human review.
+
+Usage:
+ # Build the translation index first (one-time or when .po files change)
+ python scripts/translations/build_translation_index.py
+
+ # Backfill French translations
+ python scripts/translations/backfill_po.py --lang fr
+
+ # Dry run (print what would be translated without writing)
+ python scripts/translations/backfill_po.py --lang de --dry-run
+
+ # Limit to 100 entries and use a specific model
+ python scripts/translations/backfill_po.py --lang es --limit 100 \
+ --model claude-opus-4-6
+
+Options:
+ --lang LANG ISO language code to backfill (required)
+ --batch-size N Number of strings per Claude request (default: 50)
+ --limit N Stop after translating N entries (default: unlimited)
+ --model MODEL Claude model ID (default: claude-sonnet-4-6)
+ --index PATH Path to translation_index.json (default: auto-detect)
+ --dry-run Print translations without writing to .po file
+ --fuzzy/--no-fuzzy Mark generated translations as fuzzy (default: fuzzy)
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any
+
+try:
+ import polib # type: ignore[import-untyped]
+except ImportError:
+ print("polib is required. Run: pip install polib", file=sys.stderr)
+ sys.exit(1)
+
+TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations"
+DEFAULT_INDEX = TRANSLATIONS_DIR / "translation_index.json"
+DEFAULT_MODEL = "claude-sonnet-4-6"
+DEFAULT_BATCH_SIZE = 50
+
+# Language names for the prompt, keyed by ISO code
+LANGUAGE_NAMES: dict[str, str] = {
+ "ar": "Arabic",
+ "ca": "Catalan",
+ "de": "German",
+ "es": "Spanish",
+ "fa": "Persian (Farsi)",
+ "fr": "French",
+ "it": "Italian",
+ "ja": "Japanese",
+ "ko": "Korean",
+ "mi": "Māori",
+ "nl": "Dutch",
+ "pl": "Polish",
+ "pt": "Portuguese",
+ "pt_BR": "Brazilian Portuguese",
+ "ru": "Russian",
+ "sk": "Slovak",
+ "sl": "Slovenian",
+ "tr": "Turkish",
+ "uk": "Ukrainian",
+ "zh": "Chinese (Simplified)",
+ "zh_TW": "Chinese (Traditional)",
+}
+
+
+def _lang_name(code: str) -> str:
+ return LANGUAGE_NAMES.get(code, code)
Review Comment:
Addressed in 37af05c29. The docstring was applied, but accepting the "Apply suggestion" button clobbered the line below it (the actual function body), which is why the file had a syntax error and pre-commit was failing. Restored the body and kept your docstring.
##########
scripts/translations/backfill_po.py:
##########
@@ -0,0 +1,557 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Backfill missing translations in a .po file using Claude AI.
+
+For each untranslated (empty msgstr) entry in the target language, the script
+sends the English source string along with all available translations in other
+languages to Claude as context, then writes the AI-generated translation back
+into the .po file marked as #, fuzzy for human review.
+
+Usage:
+ # Build the translation index first (one-time or when .po files change)
+ python scripts/translations/build_translation_index.py
+
+ # Backfill French translations
+ python scripts/translations/backfill_po.py --lang fr
+
+ # Dry run (print what would be translated without writing)
+ python scripts/translations/backfill_po.py --lang de --dry-run
+
+ # Limit to 100 entries and use a specific model
+ python scripts/translations/backfill_po.py --lang es --limit 100 \
+ --model claude-opus-4-6
+
+Options:
+ --lang LANG ISO language code to backfill (required)
+ --batch-size N Number of strings per Claude request (default: 50)
+ --limit N Stop after translating N entries (default: unlimited)
+ --model MODEL Claude model ID (default: claude-sonnet-4-6)
+ --index PATH Path to translation_index.json (default: auto-detect)
+ --dry-run Print translations without writing to .po file
+ --fuzzy/--no-fuzzy Mark generated translations as fuzzy (default: fuzzy)
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any
+
+try:
+ import polib # type: ignore[import-untyped]
+except ImportError:
+ print("polib is required. Run: pip install polib", file=sys.stderr)
+ sys.exit(1)
+
+TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations"
+DEFAULT_INDEX = TRANSLATIONS_DIR / "translation_index.json"
+DEFAULT_MODEL = "claude-sonnet-4-6"
+DEFAULT_BATCH_SIZE = 50
+
+# Language names for the prompt, keyed by ISO code
+LANGUAGE_NAMES: dict[str, str] = {
+ "ar": "Arabic",
+ "ca": "Catalan",
+ "de": "German",
+ "es": "Spanish",
+ "fa": "Persian (Farsi)",
+ "fr": "French",
+ "it": "Italian",
+ "ja": "Japanese",
+ "ko": "Korean",
+ "mi": "Māori",
+ "nl": "Dutch",
+ "pl": "Polish",
+ "pt": "Portuguese",
+ "pt_BR": "Brazilian Portuguese",
+ "ru": "Russian",
+ "sk": "Slovak",
+ "sl": "Slovenian",
+ "tr": "Turkish",
+ "uk": "Ukrainian",
+ "zh": "Chinese (Simplified)",
+ "zh_TW": "Chinese (Traditional)",
+}
+
+
+def _lang_name(code: str) -> str:
+ return LANGUAGE_NAMES.get(code, code)
+
+
+def _plural_key(msgid: str, msgid_plural: str) -> str:
+ return f"{msgid}\x00{msgid_plural}"
Review Comment:
Addressed in 37af05c29. The docstring was applied, but accepting the "Apply suggestion" button clobbered the line below it (the actual function body), which is why the file had a syntax error and pre-commit was failing. Restored the body and kept your docstring.
##########
scripts/translations/backfill_po.py:
##########
@@ -0,0 +1,557 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Backfill missing translations in a .po file using Claude AI.
+
+For each untranslated (empty msgstr) entry in the target language, the script
+sends the English source string along with all available translations in other
+languages to Claude as context, then writes the AI-generated translation back
+into the .po file marked as #, fuzzy for human review.
+
+Usage:
+ # Build the translation index first (one-time or when .po files change)
+ python scripts/translations/build_translation_index.py
+
+ # Backfill French translations
+ python scripts/translations/backfill_po.py --lang fr
+
+ # Dry run (print what would be translated without writing)
+ python scripts/translations/backfill_po.py --lang de --dry-run
+
+ # Limit to 100 entries and use a specific model
+ python scripts/translations/backfill_po.py --lang es --limit 100 \
+ --model claude-opus-4-6
+
+Options:
+ --lang LANG ISO language code to backfill (required)
+ --batch-size N Number of strings per Claude request (default: 50)
+ --limit N Stop after translating N entries (default: unlimited)
+ --model MODEL Claude model ID (default: claude-sonnet-4-6)
+ --index PATH Path to translation_index.json (default: auto-detect)
+ --dry-run Print translations without writing to .po file
+ --fuzzy/--no-fuzzy Mark generated translations as fuzzy (default: fuzzy)
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any
+
+try:
+ import polib # type: ignore[import-untyped]
+except ImportError:
+ print("polib is required. Run: pip install polib", file=sys.stderr)
+ sys.exit(1)
+
+TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations"
+DEFAULT_INDEX = TRANSLATIONS_DIR / "translation_index.json"
+DEFAULT_MODEL = "claude-sonnet-4-6"
+DEFAULT_BATCH_SIZE = 50
+
+# Language names for the prompt, keyed by ISO code
+LANGUAGE_NAMES: dict[str, str] = {
+ "ar": "Arabic",
+ "ca": "Catalan",
+ "de": "German",
+ "es": "Spanish",
+ "fa": "Persian (Farsi)",
+ "fr": "French",
+ "it": "Italian",
+ "ja": "Japanese",
+ "ko": "Korean",
+ "mi": "Māori",
+ "nl": "Dutch",
+ "pl": "Polish",
+ "pt": "Portuguese",
+ "pt_BR": "Brazilian Portuguese",
+ "ru": "Russian",
+ "sk": "Slovak",
+ "sl": "Slovenian",
+ "tr": "Turkish",
+ "uk": "Ukrainian",
+ "zh": "Chinese (Simplified)",
+ "zh_TW": "Chinese (Traditional)",
+}
+
+
+def _lang_name(code: str) -> str:
+ return LANGUAGE_NAMES.get(code, code)
+
+
+def _plural_key(msgid: str, msgid_plural: str) -> str:
+ return f"{msgid}\x00{msgid_plural}"
+
+
+def _is_missing(entry: polib.POEntry) -> bool:
+ """Return True for entries that need a translation."""
+ if entry.obsolete:
+ return False
+ if entry.msgid_plural:
+ return not any(v for v in entry.msgstr_plural.values())
+ return not entry.msgstr
+
+
+def _context_langs(
+ item: dict[str, Any], index: dict[str, Any], target_lang: str
+) -> list[str]:
+ """Return sorted list of language codes that have translations for this entry."""
+ key = item["index_key"]
+ if key not in index:
+ return []
+ return sorted(
+ lang for lang, val in index[key].items() if lang != target_lang and val
+ )
+
+
+def _context_count(
+ item: dict[str, Any], index: dict[str, Any], target_lang: str
+) -> int:
+ """Return the number of other-language translations available for this entry."""
+ return len(_context_langs(item, index, target_lang))
+
+
+def _render_item(
+ i: int,
+ item: dict[str, Any],
+ index: dict[str, Any],
+ target_lang: str,
+ reference_langs_sorted: list[str],
+) -> list[str]:
+ """Render one batch entry as prompt lines."""
+ lines: list[str] = []
+ ctx = _context_count(item, index, target_lang)
+ if ctx == 0:
+ lines.append(
+ f"--- [{i}] (no reference translations — translate conservatively) ---"
+ )
+ else:
+ plural = "s" if ctx != 1 else ""
+ lines.append(f"--- [{i}] ({ctx} reference translation{plural}) ---")
+ lines.append(f"English: {json.dumps(item['msgid'], ensure_ascii=False)}")
+ if item.get("msgid_plural"):
+ plural_json = json.dumps(item["msgid_plural"], ensure_ascii=False)
+ lines.append(f"English plural: {plural_json}")
+ key = item["index_key"]
+ if key in index and reference_langs_sorted:
+ for lang in reference_langs_sorted:
+ val = index[key].get(lang)
+ if val is None:
+ continue
+ if isinstance(val, dict):
+ forms = "; ".join(
+ f"[{k}] {json.dumps(v, ensure_ascii=False)}" for k, v in val.items()
+ )
+ lines.append(f"{_lang_name(lang)}: {forms}")
+ else:
+ lines.append(
+ f"{_lang_name(lang)}: {json.dumps(val, ensure_ascii=False)}"
+ )
+ lines.append("")
+ return lines
+
+
+def build_prompt(
+ target_lang: str,
+ batch: list[dict[str, Any]],
+ index: dict[str, Any],
+) -> str:
+ """Build the Claude prompt for a batch of entries."""
+ lang_name = _lang_name(target_lang)
+
+ # Collect which other languages actually have translations for this batch
+ reference_langs: set[str] = set()
+ for item in batch:
+ key = item["index_key"]
+ if key in index:
+ reference_langs.update(
+ lang for lang, val in index[key].items() if lang != target_lang and val
+ )
+ reference_langs_sorted = sorted(reference_langs)
+
+ lines: list[str] = [
+ "You are a professional translator specializing in software UI strings.",
+ f"Translate the following English strings into {lang_name} ({target_lang}).",
+ "",
+ "Rules:",
+ "- Preserve all format placeholders exactly: %(name)s, {name}, %s, %d, etc.",
+ "- Preserve HTML tags if present.",
+ "- Keep the same tone and register as the reference translations.",
+ "- For plural forms, provide translations for all plural forms"
+ " required by the language.",
+ "- Return ONLY a JSON object mapping each numeric index (as a string)"
+ " to its translation.",
+ "- Do not add any explanation, preamble, or markdown fences.",
+ "",
+ "Important: Many strings are short fragments or single words that are"
+ " ambiguous in English (e.g. 'Scale' could mean a measurement scale,"
+ " to scale an image, or fish scales). Use the translations in other"
+ " languages as your primary signal for which meaning is intended —"
+ " they collectively disambiguate the intended sense. When no"
+ " other-language translations are available for an entry, translate"
+ " conservatively based on the most common meaning in a data"
+ " visualization UI context.",
+ "",
+ ]
+
+ if reference_langs_sorted:
+ lines.append(
+ f"Reference translations are provided per string where available "
+ f"({', '.join(_lang_name(lc) for lc in reference_langs_sorted)})."
+ )
+ lines.append("")
+
+ lines.append("Strings to translate:")
+ lines.append("")
+
+ for i, item in enumerate(batch):
+ lines.extend(_render_item(i, item, index, target_lang, reference_langs_sorted))
+
+ if batch and batch[0].get("msgid_plural"):
+ # Add guidance on plural form counts per language
+ lines.append(
+ "Note: provide ALL plural forms required by the target language "
+ "(e.g. French needs 2, Russian needs 3, Arabic needs 6)."
+ )
+ lines.append("")
+
+ lines.append(
+ 'Expected output format: {"0": "<translation>", "1": "<translation>", ...}'
+ )
+ lines.append("(keys are the numeric indices of the strings above)")
+
+ return "\n".join(lines)
+
+
+def parse_response(text: str, batch_size: int) -> dict[int, str]:
+ """Parse the JSON object from Claude's response."""
+ # Strip any accidental markdown fences
+ text = re.sub(r"^```[^\n]*\n", "", text.strip())
+ text = re.sub(r"\n```$", "", text)
+ try:
+ raw = json.loads(text)
+ return {int(k): str(v) for k, v in raw.items() if str(k).isdigit()}
+ except json.JSONDecodeError as exc:
+ raise ValueError(
+ f"Could not parse response as JSON: {exc}\n\nResponse:\n{text}"
+ ) from exc
+
+
+def translate_batch(
+ model: str,
+ target_lang: str,
+ batch: list[dict[str, Any]],
+ index: dict[str, Any],
+) -> dict[int, str]:
+ """Send a batch of strings to Claude via `claude -p`.
+
+ Returns a dict mapping batch index to translated string.
+ """
+ claude_bin = shutil.which("claude")
+ if not claude_bin:
+ raise RuntimeError(
+ "claude CLI not found. Install Claude Code or add it to PATH."
+ )
+ prompt = build_prompt(target_lang, batch, index)
+ # claude_bin is resolved via shutil.which — not user-controlled input
+ result = subprocess.run( # noqa: S603
+ [claude_bin, "--model", model, "-p", prompt],
+ capture_output=True,
+ text=True,
+ check=False,
+ )
+ if result.returncode != 0:
+ raise RuntimeError(
+ f"claude exited with code {result.returncode}:\n{result.stderr}"
+ )
+ return parse_response(result.stdout.strip(), len(batch))
+
+
+def _apply_translation(
+ entry: polib.POEntry,
+ translation: str,
+ item: dict[str, Any],
+ model: str,
+ mark_fuzzy: bool,
+) -> None:
+ """Write a translation string into a POEntry and add attribution."""
+ if entry.msgid_plural:
+ # Model may return a JSON dict of plural forms or a plain string
+ try:
+ plural_dict = json.loads(translation)
+ if isinstance(plural_dict, dict):
+ entry.msgstr_plural = {int(k): str(v) for k, v in plural_dict.items()}
+ else:
+ for k in entry.msgstr_plural:
+ entry.msgstr_plural[k] = str(plural_dict)
+ except (json.JSONDecodeError, ValueError):
+ for k in entry.msgstr_plural:
+ entry.msgstr_plural[k] = translation
+ else:
+ entry.msgstr = translation
+
+ if mark_fuzzy and "fuzzy" not in entry.flags:
+ entry.flags.append("fuzzy")
+
+ refs = item["context_langs"]
+ refs_tag = f" [refs: {', '.join(refs)}]" if refs else " [no refs]"
+ attribution = f"Machine-translated via backfill_po.py ({model}){refs_tag}"
+ if entry.tcomment:
+ if attribution not in entry.tcomment:
+ entry.tcomment = f"{entry.tcomment}\n{attribution}"
+ else:
+ entry.tcomment = attribution
+
+
+def _build_batch_items(
+ entries: list[polib.POEntry],
+ index: dict[str, Any],
+ lang: str,
+) -> list[dict[str, Any]]:
+ """Convert a list of POEntries into the dict format used by translate_batch."""
+ items: list[dict[str, Any]] = []
+ for entry in entries:
+ if entry.msgid_plural:
+ item: dict[str, Any] = {
+ "msgid": entry.msgid,
+ "msgid_plural": entry.msgid_plural,
+ "index_key": _plural_key(entry.msgid, entry.msgid_plural),
+ "is_plural": True,
+ }
+ else:
+ item = {
+ "msgid": entry.msgid,
+ "index_key": entry.msgid,
+ "is_plural": False,
+ }
+ item["context_langs"] = _context_langs(item, index, lang)
+ item["context_count"] = len(item["context_langs"])
+ items.append(item)
+ return items
+
+
+def _process_batches(
+ missing: list[polib.POEntry],
+ index: dict[str, Any],
+ lang: str,
+ batch_size: int,
+ model: str,
+ dry_run: bool,
+ mark_fuzzy: bool,
+) -> tuple[int, int]:
+ """Translate missing entries in batches. Returns (translated, failed) counts."""
+ translated_count = 0
+ failed_count = 0
+ for batch_start in range(0, len(missing), batch_size):
+ batch_entries = missing[batch_start : batch_start + batch_size]
+ batch_items = _build_batch_items(batch_entries, index, lang)
+ end = min(batch_start + batch_size, len(missing))
+ print(
+ f" Translating entries {batch_start + 1}–{end} of {len(missing)} …",
+ file=sys.stderr,
+ )
+ try:
+ translations = translate_batch(model, lang, batch_items, index)
+ except (ValueError, RuntimeError) as exc:
+ print(f" ERROR in batch starting at {batch_start}: {exc}", file=sys.stderr)
+ failed_count += len(batch_entries)
+ continue
+ for i, entry in enumerate(batch_entries):
+ translation = translations.get(i)
+ if translation is None:
+ print(
+ f" WARNING: no translation returned for index {i} "
+ f"(msgid: {entry.msgid[:60]!r})",
+ file=sys.stderr,
+ )
+ failed_count += 1
+ continue
+ if dry_run:
+ ctx = batch_items[i]["context_count"]
+ ctx_tag = f" [ctx:{ctx}]" if ctx < 3 else ""
+ print(
+ f" [{lang}]{ctx_tag} {entry.msgid[:60]!r} → {translation[:60]!r}"
+ )
+ else:
+ _apply_translation(
+ entry, translation, batch_items[i], model, mark_fuzzy
+ )
+ translated_count += 1
+ return translated_count, failed_count
+
+
+def backfill(
+ lang: str,
+ *,
+ batch_size: int = DEFAULT_BATCH_SIZE,
+ limit: int | None = None,
+ min_context: int = 0,
+ model: str = DEFAULT_MODEL,
+ index_path: Path = DEFAULT_INDEX,
+ dry_run: bool = False,
+ mark_fuzzy: bool = True,
+) -> None:
+ po_path = TRANSLATIONS_DIR / lang / "LC_MESSAGES" / "messages.po"
+ if not po_path.exists():
+ print(f"No .po file found for language '{lang}': {po_path}", file=sys.stderr)
+ sys.exit(1)
+ if not index_path.exists():
+ print(
+ f"Translation index not found at {index_path}.\n"
+ "Run: python scripts/translations/build_translation_index.py",
+ file=sys.stderr,
+ )
+ sys.exit(1)
+
+ print("Loading translation index …", file=sys.stderr)
+ with open(index_path, encoding="utf-8") as f:
+ index: dict[str, Any] = json.load(f)
+
+ print(f"Loading {po_path} …", file=sys.stderr)
+ cat = polib.pofile(str(po_path))
+
+ missing: list[polib.POEntry] = [e for e in cat if e.msgid and _is_missing(e)]
+ print(f"Found {len(missing)} untranslated entries for '{lang}'.", file=sys.stderr)
+
+ if min_context > 0:
+ before = len(missing)
+ missing = [
+ e
+ for e in missing
+ if _context_count(
+ {
+ "index_key": (
+ _plural_key(e.msgid, e.msgid_plural)
+ if e.msgid_plural
+ else e.msgid
+ )
+ },
+ index,
+ lang,
+ )
+ >= min_context
+ ]
+ skipped = before - len(missing)
+ print(
+ f"Skipping {skipped} entries with fewer than {min_context} reference "
+ f"translation(s) (use --min-context 0 to include them).",
+ file=sys.stderr,
+ )
+
+ if limit is not None:
+ missing = missing[:limit]
+ print(f"Limiting to {limit} entries.", file=sys.stderr)
+
+ if not missing:
+ print("Nothing to do.", file=sys.stderr)
+ return
+
+ translated_count, failed_count = _process_batches(
+ missing, index, lang, batch_size, model, dry_run, mark_fuzzy
+ )
+
+ if not dry_run and translated_count > 0:
+ print(f"\nSaving {po_path} …", file=sys.stderr)
+ cat.save()
+
+ print(
+ f"\nDone. Translated: {translated_count}, Failed/skipped: {failed_count}.",
+ file=sys.stderr,
+ )
+ if not dry_run and translated_count > 0:
+ print(
+ f"Translations written to {po_path} (marked #, fuzzy for review).",
+ file=sys.stderr,
+ )
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(
Review Comment:
Addressed in 37af05c29. The docstring is kept; the function body that was accidentally replaced by the suggestion has been restored.
##########
scripts/translations/backfill_po.py:
##########
@@ -0,0 +1,557 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Backfill missing translations in a .po file using Claude AI.
+
+For each untranslated (empty msgstr) entry in the target language, the script
+sends the English source string along with all available translations in other
+languages to Claude as context, then writes the AI-generated translation back
+into the .po file marked as #, fuzzy for human review.
+
+Usage:
+ # Build the translation index first (one-time or when .po files change)
+ python scripts/translations/build_translation_index.py
+
+ # Backfill French translations
+ python scripts/translations/backfill_po.py --lang fr
+
+ # Dry run (print what would be translated without writing)
+ python scripts/translations/backfill_po.py --lang de --dry-run
+
+ # Limit to 100 entries and use a specific model
+ python scripts/translations/backfill_po.py --lang es --limit 100 \
+ --model claude-opus-4-6
+
+Options:
+ --lang LANG ISO language code to backfill (required)
+ --batch-size N Number of strings per Claude request (default: 50)
+ --limit N Stop after translating N entries (default: unlimited)
+ --model MODEL Claude model ID (default: claude-sonnet-4-6)
+ --index PATH Path to translation_index.json (default: auto-detect)
+ --dry-run Print translations without writing to .po file
+ --fuzzy/--no-fuzzy Mark generated translations as fuzzy (default: fuzzy)
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any
+
+try:
+ import polib # type: ignore[import-untyped]
+except ImportError:
+ print("polib is required. Run: pip install polib", file=sys.stderr)
+ sys.exit(1)
+
+TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations"
+DEFAULT_INDEX = TRANSLATIONS_DIR / "translation_index.json"
+DEFAULT_MODEL = "claude-sonnet-4-6"
+DEFAULT_BATCH_SIZE = 50
+
+# Language names for the prompt, keyed by ISO code
+LANGUAGE_NAMES: dict[str, str] = {
+ "ar": "Arabic",
+ "ca": "Catalan",
+ "de": "German",
+ "es": "Spanish",
+ "fa": "Persian (Farsi)",
+ "fr": "French",
+ "it": "Italian",
+ "ja": "Japanese",
+ "ko": "Korean",
+ "mi": "Māori",
+ "nl": "Dutch",
+ "pl": "Polish",
+ "pt": "Portuguese",
+ "pt_BR": "Brazilian Portuguese",
+ "ru": "Russian",
+ "sk": "Slovak",
+ "sl": "Slovenian",
+ "tr": "Turkish",
+ "uk": "Ukrainian",
+ "zh": "Chinese (Simplified)",
+ "zh_TW": "Chinese (Traditional)",
+}
+
+
+def _lang_name(code: str) -> str:
+ return LANGUAGE_NAMES.get(code, code)
+
+
+def _plural_key(msgid: str, msgid_plural: str) -> str:
+ return f"{msgid}\x00{msgid_plural}"
+
+
+def _is_missing(entry: polib.POEntry) -> bool:
+ """Return True for entries that need a translation."""
+ if entry.obsolete:
+ return False
+ if entry.msgid_plural:
+ return not any(v for v in entry.msgstr_plural.values())
+ return not entry.msgstr
+
+
+def _context_langs(
+ item: dict[str, Any], index: dict[str, Any], target_lang: str
+) -> list[str]:
+ """Return sorted list of language codes that have translations for this entry."""
+ key = item["index_key"]
+ if key not in index:
+ return []
+ return sorted(
+ lang for lang, val in index[key].items() if lang != target_lang and val
+ )
+
+
+def _context_count(
+ item: dict[str, Any], index: dict[str, Any], target_lang: str
+) -> int:
+ """Return the number of other-language translations available for this entry."""
+ return len(_context_langs(item, index, target_lang))
+
+
+def _render_item(
+ i: int,
+ item: dict[str, Any],
+ index: dict[str, Any],
+ target_lang: str,
+ reference_langs_sorted: list[str],
+) -> list[str]:
+ """Render one batch entry as prompt lines."""
+ lines: list[str] = []
+ ctx = _context_count(item, index, target_lang)
+ if ctx == 0:
+ lines.append(
+ f"--- [{i}] (no reference translations — translate conservatively) ---"
+ )
+ else:
+ plural = "s" if ctx != 1 else ""
+ lines.append(f"--- [{i}] ({ctx} reference translation{plural}) ---")
+ lines.append(f"English: {json.dumps(item['msgid'], ensure_ascii=False)}")
+ if item.get("msgid_plural"):
+ plural_json = json.dumps(item["msgid_plural"], ensure_ascii=False)
+ lines.append(f"English plural: {plural_json}")
+ key = item["index_key"]
+ if key in index and reference_langs_sorted:
+ for lang in reference_langs_sorted:
+ val = index[key].get(lang)
+ if val is None:
+ continue
+ if isinstance(val, dict):
+ forms = "; ".join(
+ f"[{k}] {json.dumps(v, ensure_ascii=False)}" for k, v in val.items()
+ )
+ lines.append(f"{_lang_name(lang)}: {forms}")
+ else:
+ lines.append(
+ f"{_lang_name(lang)}: {json.dumps(val, ensure_ascii=False)}"
+ )
+ lines.append("")
+ return lines
+
+
+def build_prompt(
+ target_lang: str,
+ batch: list[dict[str, Any]],
+ index: dict[str, Any],
+) -> str:
+ """Build the Claude prompt for a batch of entries."""
+ lang_name = _lang_name(target_lang)
+
+ # Collect which other languages actually have translations for this batch
+ reference_langs: set[str] = set()
+ for item in batch:
+ key = item["index_key"]
+ if key in index:
+ reference_langs.update(
+ lang for lang, val in index[key].items() if lang != target_lang and val
+ )
+ reference_langs_sorted = sorted(reference_langs)
+
+ lines: list[str] = [
+ "You are a professional translator specializing in software UI strings.",
+ f"Translate the following English strings into {lang_name} ({target_lang}).",
+ "",
+ "Rules:",
+ "- Preserve all format placeholders exactly: %(name)s, {name}, %s, %d, etc.",
+ "- Preserve HTML tags if present.",
+ "- Keep the same tone and register as the reference translations.",
+ "- For plural forms, provide translations for all plural forms"
+ " required by the language.",
+ "- Return ONLY a JSON object mapping each numeric index (as a string)"
+ " to its translation.",
+ "- Do not add any explanation, preamble, or markdown fences.",
+ "",
+ "Important: Many strings are short fragments or single words that are"
+ " ambiguous in English (e.g. 'Scale' could mean a measurement scale,"
+ " to scale an image, or fish scales). Use the translations in other"
+ " languages as your primary signal for which meaning is intended —"
+ " they collectively disambiguate the intended sense. When no"
+ " other-language translations are available for an entry, translate"
+ " conservatively based on the most common meaning in a data"
+ " visualization UI context.",
+ "",
+ ]
+
+ if reference_langs_sorted:
+ lines.append(
+ f"Reference translations are provided per string where available "
+ f"({', '.join(_lang_name(lc) for lc in reference_langs_sorted)})."
+ )
+ lines.append("")
+
+ lines.append("Strings to translate:")
+ lines.append("")
+
+ for i, item in enumerate(batch):
+ lines.extend(_render_item(i, item, index, target_lang, reference_langs_sorted))
+
+ if batch and batch[0].get("msgid_plural"):
+ # Add guidance on plural form counts per language
+ lines.append(
+ "Note: provide ALL plural forms required by the target language "
+ "(e.g. French needs 2, Russian needs 3, Arabic needs 6)."
+ )
+ lines.append("")
+
+ lines.append(
+ 'Expected output format: {"0": "<translation>", "1": "<translation>", ...}'
+ )
+ lines.append("(keys are the numeric indices of the strings above)")
+
+ return "\n".join(lines)
+
+
+def parse_response(text: str, batch_size: int) -> dict[int, str]:
+ """Parse the JSON object from Claude's response."""
+ # Strip any accidental markdown fences
+ text = re.sub(r"^```[^\n]*\n", "", text.strip())
+ text = re.sub(r"\n```$", "", text)
+ try:
+ raw = json.loads(text)
+ return {int(k): str(v) for k, v in raw.items() if str(k).isdigit()}
+ except json.JSONDecodeError as exc:
+ raise ValueError(
+ f"Could not parse response as JSON: {exc}\n\nResponse:\n{text}"
+ ) from exc
+
+
+def translate_batch(
+ model: str,
+ target_lang: str,
+ batch: list[dict[str, Any]],
+ index: dict[str, Any],
+) -> dict[int, str]:
+ """Send a batch of strings to Claude via `claude -p`.
+
+ Returns a dict mapping batch index to translated string.
+ """
+ claude_bin = shutil.which("claude")
+ if not claude_bin:
+ raise RuntimeError(
+ "claude CLI not found. Install Claude Code or add it to PATH."
+ )
+ prompt = build_prompt(target_lang, batch, index)
+ # claude_bin is resolved via shutil.which — not user-controlled input
+ result = subprocess.run( # noqa: S603
+ [claude_bin, "--model", model, "-p", prompt],
+ capture_output=True,
+ text=True,
+ check=False,
+ )
+ if result.returncode != 0:
+ raise RuntimeError(
+ f"claude exited with code {result.returncode}:\n{result.stderr}"
+ )
+ return parse_response(result.stdout.strip(), len(batch))
+
+
+def _apply_translation(
+ entry: polib.POEntry,
+ translation: str,
+ item: dict[str, Any],
+ model: str,
+ mark_fuzzy: bool,
+) -> None:
+ """Write a translation string into a POEntry and add attribution."""
+ if entry.msgid_plural:
+ # Model may return a JSON dict of plural forms or a plain string
+ try:
+ plural_dict = json.loads(translation)
+ if isinstance(plural_dict, dict):
+ entry.msgstr_plural = {int(k): str(v) for k, v in plural_dict.items()}
+ else:
+ for k in entry.msgstr_plural:
+ entry.msgstr_plural[k] = str(plural_dict)
+ except (json.JSONDecodeError, ValueError):
+ for k in entry.msgstr_plural:
+ entry.msgstr_plural[k] = translation
+ else:
+ entry.msgstr = translation
+
+ if mark_fuzzy and "fuzzy" not in entry.flags:
+ entry.flags.append("fuzzy")
+
+ refs = item["context_langs"]
+ refs_tag = f" [refs: {', '.join(refs)}]" if refs else " [no refs]"
+ attribution = f"Machine-translated via backfill_po.py ({model}){refs_tag}"
+ if entry.tcomment:
+ if attribution not in entry.tcomment:
+ entry.tcomment = f"{entry.tcomment}\n{attribution}"
+ else:
+ entry.tcomment = attribution
+
+
+def _build_batch_items(
+ entries: list[polib.POEntry],
+ index: dict[str, Any],
+ lang: str,
+) -> list[dict[str, Any]]:
+ """Convert a list of POEntries into the dict format used by translate_batch."""
+ items: list[dict[str, Any]] = []
+ for entry in entries:
+ if entry.msgid_plural:
+ item: dict[str, Any] = {
+ "msgid": entry.msgid,
+ "msgid_plural": entry.msgid_plural,
+ "index_key": _plural_key(entry.msgid, entry.msgid_plural),
+ "is_plural": True,
+ }
+ else:
+ item = {
+ "msgid": entry.msgid,
+ "index_key": entry.msgid,
+ "is_plural": False,
+ }
+ item["context_langs"] = _context_langs(item, index, lang)
+ item["context_count"] = len(item["context_langs"])
+ items.append(item)
+ return items
+
+
+def _process_batches(
+ missing: list[polib.POEntry],
+ index: dict[str, Any],
+ lang: str,
+ batch_size: int,
+ model: str,
+ dry_run: bool,
+ mark_fuzzy: bool,
+) -> tuple[int, int]:
+ """Translate missing entries in batches. Returns (translated, failed) counts."""
+ translated_count = 0
+ failed_count = 0
+ for batch_start in range(0, len(missing), batch_size):
+ batch_entries = missing[batch_start : batch_start + batch_size]
+ batch_items = _build_batch_items(batch_entries, index, lang)
+ end = min(batch_start + batch_size, len(missing))
+ print(
+ f" Translating entries {batch_start + 1}–{end} of {len(missing)} …",
+ file=sys.stderr,
+ )
+ try:
+ translations = translate_batch(model, lang, batch_items, index)
+ except (ValueError, RuntimeError) as exc:
+ print(f" ERROR in batch starting at {batch_start}: {exc}", file=sys.stderr)
+ failed_count += len(batch_entries)
+ continue
+ for i, entry in enumerate(batch_entries):
+ translation = translations.get(i)
+ if translation is None:
+ print(
+ f" WARNING: no translation returned for index {i} "
+ f"(msgid: {entry.msgid[:60]!r})",
+ file=sys.stderr,
+ )
+ failed_count += 1
+ continue
+ if dry_run:
+ ctx = batch_items[i]["context_count"]
+ ctx_tag = f" [ctx:{ctx}]" if ctx < 3 else ""
+ print(
+ f" [{lang}]{ctx_tag} {entry.msgid[:60]!r} → {translation[:60]!r}"
+ )
+ else:
+ _apply_translation(
+ entry, translation, batch_items[i], model, mark_fuzzy
+ )
+ translated_count += 1
+ return translated_count, failed_count
+
+
+def backfill(
+ lang: str,
+ *,
+ batch_size: int = DEFAULT_BATCH_SIZE,
+ limit: int | None = None,
+ min_context: int = 0,
+ model: str = DEFAULT_MODEL,
+ index_path: Path = DEFAULT_INDEX,
+ dry_run: bool = False,
+ mark_fuzzy: bool = True,
+) -> None:
Review Comment:
Addressed in 37af05c29. The docstring is kept; the function body that was accidentally replaced by the suggestion has been restored.
##########
scripts/translations/build_translation_index.py:
##########
@@ -0,0 +1,151 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Build a cross-language translation index from all .po files.
+
+Outputs a JSON file structured as:
+ {
+ "<msgid>": {
+ "<lang>": "<translated string or null>",
+ ...
+ },
+ ...
+ }
+
+For plural entries the key is "<msgid>\x00<msgid_plural>" and the value
+is a dict mapping lang -> {0: "...", 1: "..."} (or null if untranslated).
+
+Usage:
+ python scripts/translations/build_translation_index.py
+ python scripts/translations/build_translation_index.py \
+ --translations-dir superset/translations \
+ --output /tmp/translation_index.json
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from pathlib import Path
+from typing import Any
+
+try:
+ import polib # type: ignore[import-untyped]
+except ImportError:
+ print("polib is required. Install with: pip install polib", file=sys.stderr)
+ sys.exit(1)
+
+TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations"
+DEFAULT_OUTPUT = (
+ Path(__file__).parent.parent.parent
+ / "superset"
+ / "translations"
+ / "translation_index.json"
+)
+
+
+def _is_translated(entry: polib.POEntry) -> bool:
+ """Return True if the entry has a non-empty, non-fuzzy translation."""
+ if "fuzzy" in entry.flags:
+ return False
+ if entry.msgid_plural:
+ return any(v for v in entry.msgstr_plural.values())
+ return bool(entry.msgstr)
+
+
+def _plural_key(entry: polib.POEntry) -> str:
+ return f"{entry.msgid}\x00{entry.msgid_plural}"
Review Comment:
Addressed in 37af05c29. The docstring was applied, but accepting the "Apply suggestion" button clobbered the line below it (the actual function body), which is why the file had a syntax error and pre-commit was failing. Restored the body and kept your docstring.
##########
scripts/translations/build_translation_index.py:
##########
@@ -0,0 +1,151 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Build a cross-language translation index from all .po files.
+
+Outputs a JSON file structured as:
+ {
+ "<msgid>": {
+ "<lang>": "<translated string or null>",
+ ...
+ },
+ ...
+ }
+
+For plural entries the key is "<msgid>\x00<msgid_plural>" and the value
+is a dict mapping lang -> {0: "...", 1: "..."} (or null if untranslated).
+
+Usage:
+ python scripts/translations/build_translation_index.py
+ python scripts/translations/build_translation_index.py \
+ --translations-dir superset/translations \
+ --output /tmp/translation_index.json
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from pathlib import Path
+from typing import Any
+
+try:
+ import polib # type: ignore[import-untyped]
+except ImportError:
+ print("polib is required. Install with: pip install polib", file=sys.stderr)
+ sys.exit(1)
+
+TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations"
+DEFAULT_OUTPUT = (
+ Path(__file__).parent.parent.parent
+ / "superset"
+ / "translations"
+ / "translation_index.json"
+)
+
+
+def _is_translated(entry: polib.POEntry) -> bool:
+ """Return True if the entry has a non-empty, non-fuzzy translation."""
+ if "fuzzy" in entry.flags:
+ return False
+ if entry.msgid_plural:
+ return any(v for v in entry.msgstr_plural.values())
+ return bool(entry.msgstr)
+
+
+def _plural_key(entry: polib.POEntry) -> str:
+ return f"{entry.msgid}\x00{entry.msgid_plural}"
+
+
+def build_index(translations_dir: Path) -> dict[str, Any]:
+ """Read all .po files and build a combined translation index."""
+ index: dict[str, dict[str, Any]] = {}
+
+ langs = sorted(
+ d
+ for d in os.listdir(translations_dir)
+ if (translations_dir / d / "LC_MESSAGES" / "messages.po").exists()
+ and d != "en" # en has empty msgstr by convention (source = target)
+ )
+
+ for lang in langs:
+ po_path = translations_dir / lang / "LC_MESSAGES" / "messages.po"
+ cat = polib.pofile(str(po_path))
+ for entry in cat:
+ if not entry.msgid:
+ continue # skip header entry
+
+ if entry.msgid_plural:
+ key = _plural_key(entry)
+ if key not in index:
+ index[key] = {}
+ translated = (
+ dict(entry.msgstr_plural)
+ if any(v for v in entry.msgstr_plural.values())
+ else None
+ )
+ index[key][lang] = translated
+ else:
+ key = entry.msgid
+ if key not in index:
+ index[key] = {}
+ index[key][lang] = entry.msgstr or None
+
+ # Ensure every entry has a slot for every language (null if missing)
+ for key in index:
+ for lang in langs:
+ index[key].setdefault(lang, None)
+
+ return index
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(
Review Comment:
Addressed in 37af05c29. The docstring was applied, but accepting the "Apply suggestion" button clobbered the line below it (the actual function body), which is why the file had a syntax error and pre-commit was failing. Restored the body and kept your docstring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "sfirke (via GitHub)" <gi...@apache.org>.
sfirke commented on PR #39448:
URL: https://github.com/apache/superset/pull/39448#issuecomment-4519453631
We still need a codeowner review. Great addition!
It's a little weird to see at least three different Claude agents talking to each other in this thread!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(i18n): AI-assisted translation backfill tooling + Spanish translations [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #39448:
URL: https://github.com/apache/superset/pull/39448#issuecomment-4426329721
**Caught a real bug from doing a fresh test run** — pushed fix in fe3fa946c4.
I dry-ran the script then did a small `--limit 6 --batch-size 2` real run on French to confirm the per-batch save was working. The save loop is fine, but the run surfaced a regression in plural handling: the model sometimes returns a JSON **array** for plural forms (e.g. `["form0", "form1"]`, which is a perfectly valid representation since plural forms are ordered) and `_apply_translation` only handled the dict / scalar / non-JSON-string cases — the list case fell through to `str(plural_value)`, broadcasting Python list-repr (`['form0', 'form1']`) into **both** plural slots:
```po
msgstr[0] "['Ajouté 1 nouvelle colonne au jeu de données virtuel', 'Ajouté %s nouvelles colonnes au jeu de données virtuel']"
msgstr[1] "['Ajouté 1 nouvelle colonne au jeu de données virtuel', 'Ajouté %s nouvelles colonnes au jeu de données virtuel']"
```
The committed Spanish translations escaped this — I scanned the .po and didn't find any list-repr — but it would have hit us on any future run for any language.
**Fix:**
- Extracted `_apply_plural_translation` helper. Handles dict (existing), list (new), scalar, and non-JSON-string. List path distributes by index and repeats the last form if the model returned fewer forms than the language requires (better than leaving slots blank, which would render as the raw English msgid via gettext fallback).
- The split also drops the cyclomatic complexity of `_apply_translation` back below the C901 threshold.
- 4 new regression tests: list response, list round-tripped through `parse_response`, list shorter than required forms, and empty list.
**Verified end-to-end:** with the fix, the same French entry that previously broke now writes correctly:
```po
msgstr[0] "1 nouvelle colonne ajoutée au jeu de données virtuel"
msgstr[1] "%s nouvelles colonnes ajoutées au jeu de données virtuel"
```
So @sadpandajoe — the test push was worth it; the consistency check exposed something we'd have shipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org