You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kaxil (via GitHub)" <gi...@apache.org> on 2026/09/22 11:23:08 UTC
[PR] Speed up the templated-fields static check by parsing each source file once [airflow]
kaxil opened a new pull request, #73540:
URL: https://github.com/apache/airflow/pull/73540
The `check-template-fields-valid` prek hook is the slowest hook in the CI `Static checks` job at about 146 s per pull request ([example](https://github.com/apache/airflow/actions/runs/35700940143/job/106662807685)), and it runs on every PR because `--all-files` selects it whenever provider operator files exist. Importing the 346 provider operator and sensor modules is not the cost: that takes 33 s in the CI image. The other 100 s went into parsing source over and over.
For every operator the check walks the MRO and, for each class with its own `__init__`, calls `inspect.getsource` and `ast.parse` to collect the `self.<field>` assignments. Two multipliers stacked up. `BaseOperator` and the other shared bases sit in the MRO of every operator, so they were parsed once per subclass, over a thousand times. And on Python 3.10 `inspect.getsource` on a class parses the whole module to locate the class by name, so a file like the BigQuery operators module was parsed in full once per class it defines.
The check now parses each source file once, indexes every class in it by qualified name with the same `InstanceFieldExtractor`, and caches that per file. Lookups by `__qualname__` replace both repeated parses. The first definition wins when a name is defined twice in a file, matching `inspect.getsource`. The comparison logic is unchanged, and the per-module function returns its errors instead of appending to a module-level list.
Measured inside breeze on the full module set, container startup (3 s) included:
| | Before | After |
|---|---|---|
| Full check | 136 s | 39 s |
Equivalence was checked directly: for all 1,145 distinct classes reached through operator MROs, the per-file index returns exactly what per-class `inspect.getsource` parsing returned. An injected bogus entry in an operator's `template_fields` is still reported for the operator and its subclass, with the same non-zero exit.
**Why not parallelize the imports instead.** A `ProcessPoolExecutor` over the modules was measured and rejected: it ran slower than the cached serial version, because each worker repeats the heavy shared SDK imports (Google alone is 13 s of the 33 s) and the CI image compiles source on import, so fanning out multiplies that work instead of dividing it.
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up the templated-fields static check by parsing each source file once [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #73540:
URL: https://github.com/apache/airflow/pull/73540#issuecomment-5775610809
> The `check-template-fields-valid` prek hook is the slowest hook in the CI `Static checks` job at about 146 s per pull request ([example](https://github.com/apache/airflow/actions/runs/35700940143/job/106662807685)), and it runs on every PR because `--all-files` selects it whenever provider operator files exist. Importing the 346 provider operator and sensor modules is not the cost: that takes 33 s in the CI image. The other 100 s went into parsing source over and over.
>
> For every operator the check walks the MRO and, for each class with its own `__init__`, calls `inspect.getsource` and `ast.parse` to collect the `self.<field>` assignments. Two multipliers stacked up. `BaseOperator` and the other shared bases sit in the MRO of every operator, so they were parsed once per subclass, over a thousand times. And on Python 3.10 `inspect.getsource` on a class parses the whole module to locate the class by name, so a file like the BigQuery operators module was parsed in full once per class it defines.
>
> The check now parses each source file once, indexes every class in it by qualified name with the same `InstanceFieldExtractor`, and caches that per file. Lookups by `__qualname__` replace both repeated parses. The first definition wins when a name is defined twice in a file, matching `inspect.getsource`. The comparison logic is unchanged, and the per-module function returns its errors instead of appending to a module-level list.
>
> Measured inside breeze on the full module set, container startup (3 s) included:
>
> Before After
> Full check 136 s 39 s
> Equivalence was checked directly: for all 1,145 distinct classes reached through operator MROs, the per-file index returns exactly what per-class `inspect.getsource` parsing returned. An injected bogus entry in an operator's `template_fields` is still reported for the operator and its subclass, with the same non-zero exit.
>
> **Why not parallelize the imports instead.** A `ProcessPoolExecutor` over the modules was measured and rejected: it ran slower than the cached serial version, because each worker repeats the heavy shared SDK imports (Google alone is 13 s of the 33 s) and the CI image compiles source on import, so fanning out multiplies that work instead of dividing it.
>
> * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
> * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
> * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
> * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
Thanks :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up the templated-fields static check by parsing each source file once [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil merged PR #73540:
URL: https://github.com/apache/airflow/pull/73540
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up the templated-fields static check by parsing each source file once [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #73540:
URL: https://github.com/apache/airflow/pull/73540#issuecomment-5776729538
### Backport failed to create: v3-3-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/35728784827'> Run details </a>
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>❌</td>
<td>v3-3-test</td>
<td><a href="https://github.com/apache/airflow/commit/1e87d0f02fd73bce7763c1d81225b1e6c0151fe3"><img src='https://img.shields.io/badge/Commit-1e87d0f-red' alt='Commit Link'></a></td>
</tr>
</table>
You can attempt to backport this manually by running:
```bash
cherry_picker 1e87d0f v3-3-test
```
This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.
After you have resolved the conflicts, you can continue the backport process by running:
```bash
cherry_picker --continue
```
If you don't have cherry-picker installed, see the [installation guide](https://github.com/apache/airflow/blob/main/dev/README.md#how-to-backport-pr-with-cherry-picker-cli).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org