You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2025/06/09 15:43:58 UTC
[PR] Add progress summary to translation_completeness script [airflow]
potiuk opened a new pull request, #51537:
URL: https://github.com/apache/airflow/pull/51537
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add progress summary to translation_completeness script [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #51537:
URL: https://github.com/apache/airflow/pull/51537#issuecomment-2969784821
> Do we want this script to be runnable with breeze command? Maybe be automatically commented with output on PRs that modify translations?
No need. With `uv` and `uv run` we do not need breeze to run it. For standalone scripts that do one thing, there is no neeed to have the whole breeze environment setup.
> Maybe be automatically commented with output on PRs that modify translations?
For the moment - as described in the process - it's more of a tool helpful for translators (to help with incremental adding of translations) and for release managers to see how much translation is left. As explained in the process - it's difficult to make it automaticly "fail" or otherwise signal translator that things are missing.
Because it's the "english" translation change that will immediately make all the other translations "missing" things. Things in translation will get "missing" without involvment of the translators and it requires work of all translators to "fix" them independently.
So automation in PR does not help much on PR level.
What we already have have is "pre-commit" to check if there are any TODO: left. in translation after they were added with the `--add-missing`.
But if someone figures a better way of automating it and workflow that can be applied here - yep, it could be used as part of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add progress summary to translation_completeness script [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #51537:
URL: https://github.com/apache/airflow/pull/51537#discussion_r2136320376
##########
airflow-core/src/airflow/ui/src/i18n/check_translations_completeness.py:
##########
@@ -235,6 +245,123 @@ def get_outdated_entries_for_language(
return outdated
+def count_todo_and_total(obj):
+ """Recursively count TODO: translate entries and total entries in a dict or list."""
+ todo = 0
+ total = 0
+ if isinstance(obj, dict):
+ for v in obj.values():
+ t, n = count_todo_and_total(v)
+ todo += t
+ total += n
+ elif isinstance(obj, list):
+ for v in obj:
+ t, n = count_todo_and_total(v)
+ todo += t
+ total += n
+ else:
+ total = 1
+ if isinstance(obj, str) and obj.strip().startswith("TODO: translate"):
+ todo = 1
+ return todo, total
+
+
+def print_translation_progress(console, locale_files, missing_counts, summary):
+ from rich.table import Table
+
+ tables = defaultdict(lambda: Table(show_lines=True))
+ all_files = set()
+ for lf in locale_files:
+ all_files.update(lf.files)
+ for lf in locale_files:
+ lang = lf.locale
+ table = tables[lang]
+ table.title = f"Translation Progress: {lang}"
+ table.add_column("File", style="bold cyan")
+ table.add_column("Missing", style="red")
+ table.add_column("Extra", style="yellow")
+ table.add_column("TODOs", style="magenta")
+ table.add_column("Translated", style="green")
+ table.add_column("Total", style="bold")
+ table.add_column("Percent", style="bold")
+ total_missing = 0
+ total_extra = 0
+ total_todos = 0
+ total_translated = 0
+ total_total = 0
+ for filename in sorted(all_files):
+ file_path = Path(LOCALES_DIR / lang / filename)
+ # Always get total from English version
+ en_path = Path(LOCALES_DIR / "en" / filename)
+ if en_path.exists():
+ with open(en_path) as f:
+ en_data = json.load(f)
+ file_total = sum(1 for _ in flatten_keys(en_data))
+ else:
+ file_total = 0
+ if file_path.exists():
+ with open(file_path) as f:
+ data = json.load(f)
+ file_missing = missing_counts.get(filename, {}).get(lang, 0)
+ file_extra = len(summary.get(filename, LocaleSummary({}, {})).extra_keys.get(lang, []))
+
+ # Count TODOs
+ def count_todos(obj):
+ if isinstance(obj, dict):
+ return sum(count_todos(v) for v in obj.values())
+ if isinstance(obj, list):
+ return sum(count_todos(v) for v in obj)
+ if isinstance(obj, str) and obj.strip().startswith("TODO: translate"):
+ return 1
+ return 0
+
+ file_todos = count_todos(data)
+ file_translated = file_total - file_missing
+ percent = 100 * file_translated / file_total if file_total else 100
+ style = (
+ "bold green"
+ if file_missing == 0 and file_extra == 0 and file_todos == 0
+ else (
+ "yellow" if file_missing < file_total or file_extra > 0 or file_todos > 0 else "red"
+ )
+ )
+ else:
+ file_missing = file_total
+ file_extra = len(summary.get(filename, LocaleSummary({}, {})).extra_keys.get(lang, []))
+ file_todos = 0
+ file_translated = 0
+ percent = 0
+ style = "red"
+ table.add_row(
+ f"[bold reverse]{filename}[/bold reverse]",
+ str(file_missing),
+ str(file_extra),
+ str(file_todos),
+ str(file_translated),
+ str(file_total),
+ f"{percent:.1f}%",
+ style=style,
+ )
+ total_missing += file_missing
+ total_extra += file_extra
+ total_todos += file_todos
+ total_translated += file_translated
+ total_total += file_total
+ percent = 100 * total_translated / total_total if total_total else 100
+ table.add_row(
+ "All files",
+ str(total_missing),
+ str(total_extra),
+ str(total_todos),
+ str(total_translated),
+ str(total_total),
+ f"{percent:.1f}%",
+ style="red" if percent < 100 else "bold green",
+ )
+ for _lang, table in tables.items():
Review Comment:
Feel free :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add progress summary to translation_completeness script [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #51537:
URL: https://github.com/apache/airflow/pull/51537
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add progress summary to translation_completeness script [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #51537:
URL: https://github.com/apache/airflow/pull/51537#issuecomment-2969394915
Do we want this script to be runnable with breeze command?
Maybe be automatically commented with output on PRs that modify translations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add progress summary to translation_completeness script [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #51537:
URL: https://github.com/apache/airflow/pull/51537#issuecomment-2956180065
<img width="605" alt="Screenshot 2025-06-09 at 17 44 32" src="https://github.com/user-attachments/assets/b440c96c-4371-44ed-aa13-750cf188ae37" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add progress summary to translation_completeness script [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #51537:
URL: https://github.com/apache/airflow/pull/51537#discussion_r2136280830
##########
airflow-core/src/airflow/ui/src/i18n/check_translations_completeness.py:
##########
@@ -235,6 +245,123 @@ def get_outdated_entries_for_language(
return outdated
+def count_todo_and_total(obj):
+ """Recursively count TODO: translate entries and total entries in a dict or list."""
+ todo = 0
+ total = 0
+ if isinstance(obj, dict):
+ for v in obj.values():
+ t, n = count_todo_and_total(v)
+ todo += t
+ total += n
+ elif isinstance(obj, list):
+ for v in obj:
+ t, n = count_todo_and_total(v)
+ todo += t
+ total += n
+ else:
+ total = 1
+ if isinstance(obj, str) and obj.strip().startswith("TODO: translate"):
+ todo = 1
+ return todo, total
+
+
+def print_translation_progress(console, locale_files, missing_counts, summary):
+ from rich.table import Table
+
+ tables = defaultdict(lambda: Table(show_lines=True))
+ all_files = set()
+ for lf in locale_files:
+ all_files.update(lf.files)
+ for lf in locale_files:
+ lang = lf.locale
+ table = tables[lang]
+ table.title = f"Translation Progress: {lang}"
+ table.add_column("File", style="bold cyan")
+ table.add_column("Missing", style="red")
+ table.add_column("Extra", style="yellow")
+ table.add_column("TODOs", style="magenta")
+ table.add_column("Translated", style="green")
+ table.add_column("Total", style="bold")
+ table.add_column("Percent", style="bold")
+ total_missing = 0
+ total_extra = 0
+ total_todos = 0
+ total_translated = 0
+ total_total = 0
+ for filename in sorted(all_files):
+ file_path = Path(LOCALES_DIR / lang / filename)
+ # Always get total from English version
+ en_path = Path(LOCALES_DIR / "en" / filename)
+ if en_path.exists():
+ with open(en_path) as f:
+ en_data = json.load(f)
+ file_total = sum(1 for _ in flatten_keys(en_data))
+ else:
+ file_total = 0
+ if file_path.exists():
+ with open(file_path) as f:
+ data = json.load(f)
+ file_missing = missing_counts.get(filename, {}).get(lang, 0)
+ file_extra = len(summary.get(filename, LocaleSummary({}, {})).extra_keys.get(lang, []))
+
+ # Count TODOs
+ def count_todos(obj):
+ if isinstance(obj, dict):
+ return sum(count_todos(v) for v in obj.values())
+ if isinstance(obj, list):
+ return sum(count_todos(v) for v in obj)
+ if isinstance(obj, str) and obj.strip().startswith("TODO: translate"):
+ return 1
+ return 0
+
+ file_todos = count_todos(data)
+ file_translated = file_total - file_missing
+ percent = 100 * file_translated / file_total if file_total else 100
+ style = (
+ "bold green"
+ if file_missing == 0 and file_extra == 0 and file_todos == 0
+ else (
+ "yellow" if file_missing < file_total or file_extra > 0 or file_todos > 0 else "red"
+ )
+ )
+ else:
+ file_missing = file_total
+ file_extra = len(summary.get(filename, LocaleSummary({}, {})).extra_keys.get(lang, []))
+ file_todos = 0
+ file_translated = 0
+ percent = 0
+ style = "red"
+ table.add_row(
+ f"[bold reverse]{filename}[/bold reverse]",
+ str(file_missing),
+ str(file_extra),
+ str(file_todos),
+ str(file_translated),
+ str(file_total),
+ f"{percent:.1f}%",
+ style=style,
+ )
+ total_missing += file_missing
+ total_extra += file_extra
+ total_todos += file_todos
+ total_translated += file_translated
+ total_total += file_total
+ percent = 100 * total_translated / total_total if total_total else 100
+ table.add_row(
+ "All files",
+ str(total_missing),
+ str(total_extra),
+ str(total_todos),
+ str(total_translated),
+ str(total_total),
+ f"{percent:.1f}%",
+ style="red" if percent < 100 else "bold green",
+ )
+ for _lang, table in tables.items():
Review Comment:
Maybe in the loop you should have skipped "en" :-D as this is always 100%
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org