You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "choo121600 (via GitHub)" <gi...@apache.org> on 2025/09/28 08:44:33 UTC
[PR] Add options to manage translations: sync, remove extra, and dry run [airflow]
choo121600 opened a new pull request, #56180:
URL: https://github.com/apache/airflow/pull/56180
Currently, we use the command `uv run dev/i18n/check_translations_completeness.py --language ko --add-missing`
to check translation completeness. However, this approach has limitations.
If existing content is removed and translation keys disappear from en, we must manually locate and remove those keys.
<img width="956" height="757" alt="image" src="https://github.com/user-attachments/assets/b7665912-096e-420a-96e0-ec5afe63c42a" />
To address this, I have added a new `sync` command that runs both `remove-extra` and `add-missing`.
This command automatically synchronizes removed translation keys in en across other language folders.
https://github.com/user-attachments/assets/efd91bf5-64c1-4bfe-99bf-1cbb873b22cc
https://github.com/user-attachments/assets/fed5e49e-daee-4034-a2f7-6bfa15b308aa
As a result, translation management becomes more efficient and consistent.
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add options to manage translations: sync, remove extra, and dry run [airflow]
Posted by "choo121600 (via GitHub)" <gi...@apache.org>.
choo121600 commented on code in PR #56180:
URL: https://github.com/apache/airflow/pull/56180#discussion_r2385418414
##########
dev/i18n/check_translations_completeness.py:
##########
@@ -581,11 +614,87 @@ def natural_key_sort(obj):
return obj
lang_data = natural_key_sort(lang_data)
- lang_path.parent.mkdir(parents=True, exist_ok=True)
- with open(lang_path, "w", encoding="utf-8") as f:
- json.dump(lang_data, f, ensure_ascii=False, indent=2)
- f.write("\n") # Ensure newline at the end of the file
- console.print(f"[green]Added missing translations to {lang_path}[/green]")
+
+ if dry_run:
+ console.print(f"[blue][DRY RUN] Would add missing translations to {lang_path}[/blue]")
+ else:
+ lang_path.parent.mkdir(parents=True, exist_ok=True)
+ with open(lang_path, "w", encoding="utf-8") as f:
+ json.dump(lang_data, f, ensure_ascii=False, indent=2)
+ f.write("\n") # Ensure newline at the end of the file
+ console.print(f"[green]Added missing translations to {lang_path}[/green]")
Review Comment:
Thanks for the suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add options to manage translations: sync, remove extra, and dry run [airflow]
Posted by "choo121600 (via GitHub)" <gi...@apache.org>.
choo121600 commented on code in PR #56180:
URL: https://github.com/apache/airflow/pull/56180#discussion_r2385417540
##########
dev/i18n/check_translations_completeness.py:
##########
@@ -581,11 +614,87 @@ def natural_key_sort(obj):
return obj
lang_data = natural_key_sort(lang_data)
- lang_path.parent.mkdir(parents=True, exist_ok=True)
- with open(lang_path, "w", encoding="utf-8") as f:
- json.dump(lang_data, f, ensure_ascii=False, indent=2)
- f.write("\n") # Ensure newline at the end of the file
- console.print(f"[green]Added missing translations to {lang_path}[/green]")
+
+ if dry_run:
+ console.print(f"[blue][DRY RUN] Would add missing translations to {lang_path}[/blue]")
+ else:
+ lang_path.parent.mkdir(parents=True, exist_ok=True)
+ with open(lang_path, "w", encoding="utf-8") as f:
+ json.dump(lang_data, f, ensure_ascii=False, indent=2)
+ f.write("\n") # Ensure newline at the end of the file
+ console.print(f"[green]Added missing translations to {lang_path}[/green]")
+
+
+def remove_extra_translations(
+ language: str, summary: dict[str, LocaleSummary], console: Console, dry_run: bool = False
+):
+ """
+ Remove extra translations that are not present in English.
+
+ This removes keys that exist in the target language but not in English.
+ """
+ for filename, diff in summary.items():
+ extra_keys = set(diff.extra_keys.get(language, []))
+ if not extra_keys:
+ continue
+
+ lang_path = LOCALES_DIR / language / filename
+ if not lang_path.exists():
+ continue
+
+ try:
+ lang_data = load_json(lang_path)
+ except Exception as e:
+ console.print(f"[red]Failed to load {language} file {lang_path}: {e}[/red]")
+ continue
+
+ # Helper to recursively remove extra keys
+ def remove_keys(data, prefix=""):
+ keys_to_remove = []
+ for k, v in data.items():
+ full_key = f"{prefix}.{k}" if prefix else k
+ if full_key in extra_keys:
+ keys_to_remove.append(k)
+ if dry_run:
+ console.print(f"[blue][DRY RUN] Would remove extra key: {full_key}[/blue]")
+ else:
+ console.print(f"[yellow]Removed extra key: {full_key}[/yellow]")
+ elif isinstance(v, dict):
+ remove_keys(v, full_key)
+ # Remove empty nested dicts after recursive cleanup
+ if not v:
+ keys_to_remove.append(k)
+ if dry_run:
+ console.print(
+ f"[blue][DRY RUN] Would remove empty nested dict: {full_key}[/blue]"
+ )
+ else:
+ console.print(f"[yellow]Removed empty nested dict: {full_key}[/yellow]")
+
+ if not dry_run:
+ for k in keys_to_remove:
+ del data[k]
+
+ remove_keys(lang_data)
+
+ # Only write if there were changes
+ if extra_keys:
+ if dry_run:
+ console.print(f"[blue][DRY RUN] Would remove extra translations from {lang_path}[/blue]")
+ else:
+
+ def natural_key_sort(obj):
+ if isinstance(obj, dict):
+ return {
+ k: natural_key_sort(obj[k]) for k in sorted(obj, key=lambda x: (x.lower(), x))
+ }
+ return obj
+
+ lang_data = natural_key_sort(lang_data)
+ with open(lang_path, "w", encoding="utf-8") as f:
+ json.dump(lang_data, f, ensure_ascii=False, indent=2)
+ f.write("\n") # Ensure newline at the end of the file
+ console.print(f"[green]Removed extra translations from {lang_path}[/green]")
Review Comment:
Thanks for the suggestion! đź‘Ť
I updated the function to use early return as you recommended.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add options to manage translations: sync, remove extra, and dry run [airflow]
Posted by "Programmer-RD-AI (via GitHub)" <gi...@apache.org>.
Programmer-RD-AI commented on code in PR #56180:
URL: https://github.com/apache/airflow/pull/56180#discussion_r2385376228
##########
dev/i18n/check_translations_completeness.py:
##########
@@ -581,11 +614,87 @@ def natural_key_sort(obj):
return obj
lang_data = natural_key_sort(lang_data)
- lang_path.parent.mkdir(parents=True, exist_ok=True)
- with open(lang_path, "w", encoding="utf-8") as f:
- json.dump(lang_data, f, ensure_ascii=False, indent=2)
- f.write("\n") # Ensure newline at the end of the file
- console.print(f"[green]Added missing translations to {lang_path}[/green]")
+
+ if dry_run:
+ console.print(f"[blue][DRY RUN] Would add missing translations to {lang_path}[/blue]")
+ else:
+ lang_path.parent.mkdir(parents=True, exist_ok=True)
+ with open(lang_path, "w", encoding="utf-8") as f:
+ json.dump(lang_data, f, ensure_ascii=False, indent=2)
+ f.write("\n") # Ensure newline at the end of the file
+ console.print(f"[green]Added missing translations to {lang_path}[/green]")
Review Comment:
You could simplify this branch by doing an early return inside the if dry_run. That way, you can print the dry-run message and exit immediately, instead of wrapping the actual write logic in an else.
It makes the control flow flatter and easier to read:
```python
if dry_run:
console.print(f"[blue][DRY RUN] Would add missing translations to {lang_path}[/blue]")
return
lang_path.parent.mkdir(parents=True, exist_ok=True)
with open(lang_path, "w", encoding="utf-8") as f:
json.dump(lang_data, f, ensure_ascii=False, indent=2)
f.write("\n")
console.print(f"[green]Added missing translations to {lang_path}[/green]")
```
Totally optional, but this makes the structure a bit cleaner and avoids the extra else.
##########
dev/i18n/check_translations_completeness.py:
##########
@@ -581,11 +614,87 @@ def natural_key_sort(obj):
return obj
lang_data = natural_key_sort(lang_data)
- lang_path.parent.mkdir(parents=True, exist_ok=True)
- with open(lang_path, "w", encoding="utf-8") as f:
- json.dump(lang_data, f, ensure_ascii=False, indent=2)
- f.write("\n") # Ensure newline at the end of the file
- console.print(f"[green]Added missing translations to {lang_path}[/green]")
+
+ if dry_run:
+ console.print(f"[blue][DRY RUN] Would add missing translations to {lang_path}[/blue]")
+ else:
+ lang_path.parent.mkdir(parents=True, exist_ok=True)
+ with open(lang_path, "w", encoding="utf-8") as f:
+ json.dump(lang_data, f, ensure_ascii=False, indent=2)
+ f.write("\n") # Ensure newline at the end of the file
+ console.print(f"[green]Added missing translations to {lang_path}[/green]")
+
+
+def remove_extra_translations(
+ language: str, summary: dict[str, LocaleSummary], console: Console, dry_run: bool = False
+):
+ """
+ Remove extra translations that are not present in English.
+
+ This removes keys that exist in the target language but not in English.
+ """
+ for filename, diff in summary.items():
+ extra_keys = set(diff.extra_keys.get(language, []))
+ if not extra_keys:
+ continue
+
+ lang_path = LOCALES_DIR / language / filename
+ if not lang_path.exists():
+ continue
+
+ try:
+ lang_data = load_json(lang_path)
+ except Exception as e:
+ console.print(f"[red]Failed to load {language} file {lang_path}: {e}[/red]")
+ continue
+
+ # Helper to recursively remove extra keys
+ def remove_keys(data, prefix=""):
+ keys_to_remove = []
+ for k, v in data.items():
+ full_key = f"{prefix}.{k}" if prefix else k
+ if full_key in extra_keys:
+ keys_to_remove.append(k)
+ if dry_run:
+ console.print(f"[blue][DRY RUN] Would remove extra key: {full_key}[/blue]")
+ else:
+ console.print(f"[yellow]Removed extra key: {full_key}[/yellow]")
+ elif isinstance(v, dict):
+ remove_keys(v, full_key)
+ # Remove empty nested dicts after recursive cleanup
+ if not v:
+ keys_to_remove.append(k)
+ if dry_run:
+ console.print(
+ f"[blue][DRY RUN] Would remove empty nested dict: {full_key}[/blue]"
+ )
+ else:
+ console.print(f"[yellow]Removed empty nested dict: {full_key}[/yellow]")
+
+ if not dry_run:
+ for k in keys_to_remove:
+ del data[k]
+
+ remove_keys(lang_data)
+
+ # Only write if there were changes
+ if extra_keys:
+ if dry_run:
+ console.print(f"[blue][DRY RUN] Would remove extra translations from {lang_path}[/blue]")
+ else:
+
+ def natural_key_sort(obj):
+ if isinstance(obj, dict):
+ return {
+ k: natural_key_sort(obj[k]) for k in sorted(obj, key=lambda x: (x.lower(), x))
+ }
+ return obj
+
+ lang_data = natural_key_sort(lang_data)
+ with open(lang_path, "w", encoding="utf-8") as f:
+ json.dump(lang_data, f, ensure_ascii=False, indent=2)
+ f.write("\n") # Ensure newline at the end of the file
+ console.print(f"[green]Removed extra translations from {lang_path}[/green]")
Review Comment:
Instead of wrapping the entire write block under if extra_keys:, you could do an early return when extra_keys is empty.
That way the function exits immediately in the no-op case, and the rest of the logic doesn’t need an extra nesting level. For example:
```python
if not extra_keys:
return
if dry_run:
console.print(f"[blue][DRY RUN] Would remove extra translations from {lang_path}[/blue]")
return
lang_data = _natural_key_sort(lang_data)
with open(lang_path, "w", encoding="utf-8") as f:
json.dump(lang_data, f, ensure_ascii=False, indent=2)
f.write("\n")
console.print(f"[green]Removed extra translations from {lang_path}[/green]")
```
This makes the control flow flatter and a bit easier to follow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add options to manage translations: sync, remove extra, and dry run [airflow]
Posted by "Programmer-RD-AI (via GitHub)" <gi...@apache.org>.
Programmer-RD-AI commented on code in PR #56180:
URL: https://github.com/apache/airflow/pull/56180#discussion_r2385677152
##########
dev/i18n/check_translations_completeness.py:
##########
@@ -581,12 +614,84 @@ def natural_key_sort(obj):
return obj
lang_data = natural_key_sort(lang_data)
+
+ if dry_run:
+ console.print(f"[blue][DRY RUN] Would add missing translations to {lang_path}[/blue]")
+ return
+
lang_path.parent.mkdir(parents=True, exist_ok=True)
with open(lang_path, "w", encoding="utf-8") as f:
json.dump(lang_data, f, ensure_ascii=False, indent=2)
f.write("\n") # Ensure newline at the end of the file
console.print(f"[green]Added missing translations to {lang_path}[/green]")
+def remove_extra_translations(
+ language: str, summary: dict[str, LocaleSummary], console: Console, dry_run: bool = False
+):
+ """
+ Remove extra translations that are not present in English.
+
+ This removes keys that exist in the target language but not in English.
+ """
+ for filename, diff in summary.items():
+ extra_keys = set(diff.extra_keys.get(language, []))
+ if not extra_keys:
+ continue
+
+ lang_path = LOCALES_DIR / language / filename
+ if not lang_path.exists():
+ continue
+
+ try:
+ lang_data = load_json(lang_path)
+ except Exception as e:
+ console.print(f"[red]Failed to load {language} file {lang_path}: {e}[/red]")
+ continue
+
+ # Helper to recursively remove extra keys
+ def remove_keys(data, prefix=""):
Review Comment:
```suggestion
def remove_keys(data: dict[str, Any], prefix: str=""):
```
adding types for the parameters would help
##########
dev/i18n/check_translations_completeness.py:
##########
@@ -581,12 +614,84 @@ def natural_key_sort(obj):
return obj
lang_data = natural_key_sort(lang_data)
+
+ if dry_run:
+ console.print(f"[blue][DRY RUN] Would add missing translations to {lang_path}[/blue]")
+ return
+
lang_path.parent.mkdir(parents=True, exist_ok=True)
with open(lang_path, "w", encoding="utf-8") as f:
json.dump(lang_data, f, ensure_ascii=False, indent=2)
f.write("\n") # Ensure newline at the end of the file
console.print(f"[green]Added missing translations to {lang_path}[/green]")
+def remove_extra_translations(
+ language: str, summary: dict[str, LocaleSummary], console: Console, dry_run: bool = False
+):
+ """
+ Remove extra translations that are not present in English.
+
+ This removes keys that exist in the target language but not in English.
+ """
+ for filename, diff in summary.items():
+ extra_keys = set(diff.extra_keys.get(language, []))
+ if not extra_keys:
+ continue
+
+ lang_path = LOCALES_DIR / language / filename
+ if not lang_path.exists():
+ continue
+
+ try:
+ lang_data = load_json(lang_path)
+ except Exception as e:
+ console.print(f"[red]Failed to load {language} file {lang_path}: {e}[/red]")
+ continue
+
+ # Helper to recursively remove extra keys
+ def remove_keys(data, prefix=""):
+ keys_to_remove = []
+ for k, v in data.items():
+ full_key = f"{prefix}.{k}" if prefix else k
+ if full_key in extra_keys:
+ keys_to_remove.append(k)
+ if dry_run:
+ console.print(f"[blue][DRY RUN] Would remove extra key: {full_key}[/blue]")
+ else:
+ console.print(f"[yellow]Removed extra key: {full_key}[/yellow]")
+ elif isinstance(v, dict):
+ remove_keys(v, full_key)
+ # Remove empty nested dicts after recursive cleanup
+ if not v:
+ keys_to_remove.append(k)
+ if dry_run:
+ console.print(
+ f"[blue][DRY RUN] Would remove empty nested dict: {full_key}[/blue]"
+ )
+ else:
+ console.print(f"[yellow]Removed empty nested dict: {full_key}[/yellow]")
+
+ if not dry_run:
+ for k in keys_to_remove:
+ del data[k]
+
+ remove_keys(lang_data)
+ if dry_run:
+ console.print(f"[blue][DRY RUN] Would remove extra translations from {lang_path}[/blue]")
+ return
+
+ def natural_key_sort(obj):
Review Comment:
```suggestion
def natural_key_sort(obj: Any | dict[str: Any]) -> Any | dict[str: Any]:
```
adding types for the parameters would help
##########
dev/i18n/check_translations_completeness.py:
##########
@@ -581,12 +614,84 @@ def natural_key_sort(obj):
return obj
lang_data = natural_key_sort(lang_data)
+
+ if dry_run:
+ console.print(f"[blue][DRY RUN] Would add missing translations to {lang_path}[/blue]")
+ return
+
lang_path.parent.mkdir(parents=True, exist_ok=True)
with open(lang_path, "w", encoding="utf-8") as f:
json.dump(lang_data, f, ensure_ascii=False, indent=2)
f.write("\n") # Ensure newline at the end of the file
console.print(f"[green]Added missing translations to {lang_path}[/green]")
+def remove_extra_translations(
+ language: str, summary: dict[str, LocaleSummary], console: Console, dry_run: bool = False
+):
+ """
+ Remove extra translations that are not present in English.
+
+ This removes keys that exist in the target language but not in English.
+ """
+ for filename, diff in summary.items():
+ extra_keys = set(diff.extra_keys.get(language, []))
+ if not extra_keys:
+ continue
+
+ lang_path = LOCALES_DIR / language / filename
+ if not lang_path.exists():
+ continue
Review Comment:
nit: merged conditions for cleaner flow
```python
extra_keys = set(diff.extra_keys.get(language, []))
lang_path = LOCALES_DIR / language / filename
if not extra_keys or not lang_path.exists():
continue
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add options to manage translations: sync, remove extra, and dry run [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #56180:
URL: https://github.com/apache/airflow/pull/56180#issuecomment-3524471188
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add options to manage translations: sync, remove extra, and dry run [airflow]
Posted by "choo121600 (via GitHub)" <gi...@apache.org>.
choo121600 commented on code in PR #56180:
URL: https://github.com/apache/airflow/pull/56180#discussion_r2385929234
##########
dev/i18n/check_translations_completeness.py:
##########
@@ -581,12 +614,84 @@ def natural_key_sort(obj):
return obj
lang_data = natural_key_sort(lang_data)
+
+ if dry_run:
+ console.print(f"[blue][DRY RUN] Would add missing translations to {lang_path}[/blue]")
+ return
+
lang_path.parent.mkdir(parents=True, exist_ok=True)
with open(lang_path, "w", encoding="utf-8") as f:
json.dump(lang_data, f, ensure_ascii=False, indent=2)
f.write("\n") # Ensure newline at the end of the file
console.print(f"[green]Added missing translations to {lang_path}[/green]")
+def remove_extra_translations(
+ language: str, summary: dict[str, LocaleSummary], console: Console, dry_run: bool = False
+):
+ """
+ Remove extra translations that are not present in English.
+
+ This removes keys that exist in the target language but not in English.
+ """
+ for filename, diff in summary.items():
+ extra_keys = set(diff.extra_keys.get(language, []))
+ if not extra_keys:
+ continue
+
+ lang_path = LOCALES_DIR / language / filename
+ if not lang_path.exists():
+ continue
Review Comment:
Thanksđź‘Ť
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add options to manage translations: sync, remove extra, and dry run [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #56180: Add options to manage translations: sync, remove extra, and dry run
URL: https://github.com/apache/airflow/pull/56180
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org