You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kaxil (via GitHub)" <gi...@apache.org> on 2026/09/21 18:14:50 UTC
[PR] Speed up documentation build and spellcheck in CI [airflow]
kaxil opened a new pull request, #73477:
URL: https://github.com/apache/airflow/pull/73477
The two documentation jobs in CI (`Build documentation (--docs-only)` and `(--spellcheck-only)`) take 25 to 35 minutes on `main`, and the spellcheck job took 93 minutes on a PR that adds a documentation page. This PR cuts the build itself by about 2.5x and removes the spellcheck job's worst case, without changing what gets built or checked.
Measured on the same machine with the full set of 115 packages and 4 parallel jobs (the CI runner configuration):
| | Before | After |
|---|---|---|
| Full `--docs-only` build, wall time | 645 s | 214 s |
| CPU time across the 4 workers | 41.6 min | 14.2 min |
| Median package | 17 s | 2 s |
| 16 small providers, wall time | 66 s | 36 s |
The build is now bound by the single largest package (google, 212 s of the 214 s); everything else fits around it on the other three workers.
Output is unchanged: the generated HTML for a provider is byte-identical to the previous builder's (only the pagefind index file names differ, they are content hashes).
## Design rationale
**Where the time went.** Per-package timings from the CI log show 98 of the 115 packages take under 40 s each but together cost 48 of the job's 73 CPU-minutes. Profiling one small provider: about 20 of its 22 seconds are fixed startup that has nothing to do with its own documentation. autoapi's astroid parsing of the `airflow`/`airflow.sdk` modules every provider imports (about 8.5 s), parsing and schema-validating all 108 `provider.yaml` files (about 3.4 s, done two or three times per build), importing `airflow` (1.4 s), and loading about 125 intersphinx inventories (1 s). Every package paid this in its own `sphinx-build` subprocess.
**Sphinx now runs in-process inside each pool worker** (`sphinx.cmd.build.build_main` instead of `subprocess.run`), so a worker pays that startup once and then reuses the astroid parse cache, the imported `airflow`, the parsed `provider.yaml` files and the parsed inventories for the roughly 29 packages it builds. This is the same mode `sphinx-autobuild` and Sphinx's own test suite use, and Sphinx resets the docutils registries between apps. Three things had to be handled for it to be safe:
- The per-package `conf.py` files mutate lists they import from `docs.utils.conf_constants` (`extensions.append`, `autoapi_ignore.extend`), so those modules are dropped from `sys.modules` before each build and re-executed fresh. Otherwise provider B inherits provider A's ignore patterns, including B's own package directory.
- Redirecting stdout/stderr straight to the per-package log file breaks the next build: docutils' `Reporter` inside sphinx-argparse's nested parser keeps the stream it first saw, and once that file is closed the `fab` and `edge3` `cli-ref` pages fail with "I/O operation on closed file". Output goes through a proxy stream that lives as long as the worker and only switches its target.
- The astroid cache is trimmed to drop the built package's own modules after each build, and nothing else. Deciding what to keep on the worker's resident memory does not work (a worker that built google stays at 3 GB because of the modules autodoc imported, so a memory check fires after every later build and the cache is never warm), and dropping third-party modules costs most of the gain because `airflow.sdk` pulls in attrs, pydantic and sqlalchemy that every provider then re-parses. The ASTs themselves are cheap: clearing the whole cache after an amazon build freed under 100 MB.
`sphinx-autobuild` stays a subprocess (it is a long-running server), and the old 30-minute per-build timeout is kept through `SIGALRM`.
**The spellcheck job's 93 minutes were three full passes.** The spelling builder inherits Sphinx's write phase, so it resolves cross-references and fails under `-W` on any label or page that is new in the PR, but it never writes an `objects.inv`. The docs build never hits this because the html builder writes inventories that later packages pick up. On the spellcheck side the failure triggered `rebuild_all_packages=True`: html for all 115 packages, then spelling for all 115 again. In the run linked below, one package (the providers summary) failed on the `operators/llm_batch` page this PR's neighbour added, and the job spent 57 of its 93 minutes rebuilding everything else. The spelling build now writes an inventory too (with the URIs the html build would produce), `airflow_intersphinx` prefers it over the downloaded one, and a retry only rebuilds the packages that failed, in the mode the run was started with. The `A -> B -> C` chain the extra passes were for is still covered by two
retry passes, which are no-ops when nothing failed.
**Scheduling.** Packages are handed out heaviest first with `imap_unordered(chunksize=1)`; `pool.map` pre-assigned fixed chunks of 8, which is why `task-sdk`, `helm-chart` and `docker-stack` were always the last three to start.
**Runner disk cleanup.** The docs job passes `free-up-disk-space: false` to the shared prepare action (new input, default unchanged). The step removes pre-installed tooling for 2 minutes 43 seconds; the runner has 93 GB free before it and the docs job only needs the 8 GB CI image.
`provider_yaml_utils` also stops re-validating the JSON schema on every one of the 108 files and uses the C YAML loader (2.6x faster per load); this matters less now that a worker loads once, but it is what every single-package local build pays.
## Gotchas
- A worker's resident memory now stays at the peak of the largest package it built (about 3 GB after google) instead of dropping between packages. That peak is the same the previous per-package subprocess reached; what changes is that it persists for the rest of the run. With 4 workers on a 16 GB runner this leaves headroom, and the memory is held by imported modules, not by anything the builder could release.
- The intersphinx reuse wraps a private Sphinx function (`sphinx.ext.intersphinx._load._fetch_inventory`). It exists in Sphinx 8.x, which is what CI's default Python 3.10 resolves from `uv.lock`; Sphinx 9 (Python 3.11+) split fetching from parsing, so there the wrapper is a guarded no-op, its tests skip, and each package spends about a second re-parsing inventories as before. Everything else in this PR is version-independent.
Run that prompted this: https://github.com/apache/airflow/actions/runs/35612510405 (docs-only 29 min, spellcheck 93 min).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #73477:
URL: https://github.com/apache/airflow/pull/73477#discussion_r4068627780
##########
devel-common/src/docs/build_docs.py:
##########
@@ -693,52 +729,19 @@ def build_docs(
all_spelling_errors.update(package_spelling_errors)
if not one_pass_only:
- # Build documentation for some packages again if it can help them.
- package_build_errors = retry_building_docs_if_needed(
- all_build_errors=all_build_errors,
- all_spelling_errors=all_spelling_errors,
- autobuild=autobuild,
- docs_only=docs_only,
- jobs=jobs,
- verbose=verbose,
- package_build_errors=package_build_errors,
- originally_built_packages=packages_to_build,
- # If spellchecking fails, we need to rebuild all packages first in case some references
- # are broken between packages
- rebuild_all_packages=spellcheck_only,
Review Comment:
This is what resulted in (infrequent) long spellcheck builds in edge cases,because when it failed once during spellchecking, you had to rebuild it **again** and this time all packages and not "spellcheck_only", because you did not know which package needs rebuilding to re-create inventory used.
Spellcheck builds do **not** recreate inventories. This is the problem. So if you run "spellcheck-only" build and it fails because of missing new method in another package, you need to effectively disable "spellcheck-only" - otherwise you will never have the inventory created with the new method.
But at this point you are also really guessing it - because it might also be genuine documentation error, not only inventory issue (for example typo). You really genuinly do not know here. Note - we **only** attempt that "rebuild" when there are errors indicating that **likely** there are new objects that need new inventory (There are few types of error that indicate that this might be the case). Some errors will just not be retried - the "rebuild if needed" checks the errors and will not attempt to rebuild if errors are of a different kind - it will rebuild only if all errors are "missing thing".
That **migh** result in some cases indeed the 4 th build was needed to get the docs green and you need switch from doing spellcheck only - because you need to rebuild inventories.
Imagine the case A -> B -> C (i.e. package C adding new thing, used by B and package A using new thing added in B). An example (realistic and sometimes happening) case would be (but this is extremely rare though) - adding common provider method used by SSH provider that adds new method that is used by SFTP provider. Yes. Happened.
This led to 4 (!) builds needed - one spellcheck only, and 3 doc + spellcheck builds:
1) spellcheck build -> failing (and you guess - because of new method added).
2) doc + spellcheck build of everything to rebuild inventory -> failing A,B if they were built before C . For spelcheck we need to build **all** packages not only the failing one, because spellcheck build does not produce new inventory, it only does spellcheck, so we also need to build C, to get the inventory with new methods.And this needs to be really "all packages" build here in case of spellcheck because we need to have all inventories locally - even if some of the packages failed. For example even if C spellcheck worked in 1) - there is missing inventory for C. So we really need to build **everything** here.
3) repeatebuild of A, B (those two failed. if A is build before B it will fail again. Again that one needs to be "full" build because we need also B inventory to be build.
4) build of A.
We could avoid 4 if 3) is done sequentially and if we build it in the right sequence B->A. But for that we would need to sort the remaining builds topologically.
Another option in this case is to switch such PR that adds several things into series of PRs where you knowingly add things in the right order. But that is kinda superpower to know that this is the case and figure out the right sequence and split the PR in the right way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #73477:
URL: https://github.com/apache/airflow/pull/73477#issuecomment-5773367697
> I got to the deep layers of memory and recalled the errors that led to the weird (but I think needed) up to 4 rebuilds scenarios needed sometimes @kaxil -> I am afraid the way it is simplified here will make some edge (and some not-that-edge) cases of inter-package dependent PRs to fail.
>
> The issue is that:
>
> a) spellcheck-only builds do not create inventories when they fail (!). They only use existing ones. This is why they are faster in general.
>
> b) when you run PR you cannot update inventories in "S3" - what you have there is result of last successful main build
>
> c) when you have PRs that modifies several providars and they add new methods that are used in other providers (almost always when you modify some common provideds) -> you **need** new inventories from the providers you depend on even if you are just doing spellchecking.
>
> d) we have no topological sort and no way to hold building new providers until provider dependencies that add new methods build their inventories.
>
> e) when you have unrelated errors and they prevent from creating inventories, all you really see in the output are "missing things" which are not really missing.
>
> See the detailed scenario in the inline comment.
>
> I am sure we **could** optimize it more but it's not as easy as just repeating spellcheck buit
Replied inline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #73477:
URL: https://github.com/apache/airflow/pull/73477#discussion_r4070085029
##########
devel-common/src/docs/build_docs.py:
##########
@@ -693,52 +729,19 @@ def build_docs(
all_spelling_errors.update(package_spelling_errors)
if not one_pass_only:
- # Build documentation for some packages again if it can help them.
- package_build_errors = retry_building_docs_if_needed(
- all_build_errors=all_build_errors,
- all_spelling_errors=all_spelling_errors,
- autobuild=autobuild,
- docs_only=docs_only,
- jobs=jobs,
- verbose=verbose,
- package_build_errors=package_build_errors,
- originally_built_packages=packages_to_build,
- # If spellchecking fails, we need to rebuild all packages first in case some references
- # are broken between packages
- rebuild_all_packages=spellcheck_only,
Review Comment:
> I reproduced exactly that locally with --spellcheck-only ftp sftp: a new .. _ftp-brand-new-label: in the ftp docs, referenced with :ref: from the sftp docs (sftp is heavier, so it is scheduled first):
Make another error in ftp (bad format etc.) and see what happens :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #73477:
URL: https://github.com/apache/airflow/pull/73477#issuecomment-5771754859
I got to the deep layers of memory and recalled the errors that led to the weird (but I think needed) up to 4 rebuilds scenarios needed sometimes @kaxil -> I am afraid the way it is simplified here will make some edge (and some not-that-edge) cases of inter-package dependent PRs to fail.
The issue is that:
a) spellcheck-only builds do not create inventories (!). They only use existing ones. This is why they are faster in general.
b) when you run PR you cannot update inventories in "S3" - what you have there is result of last successful main build
c) when you have PRs that modifies several providars and they add new methods that are used in other providers (almost always when you modify some common provideds) -> you **need** new inventories from the providers you depend on even if you are just doing spellchecking.
d) we have no topological sort and no way to hold building new providers until provider dependencies that add new methods build their inventories.
e) when you have unrelated errors and they prevent from creating inventories, all you really see in the output are "missing things" which are not really missing.
See the detailed scenario in the inline comment.
I am sure we **could** optimize it more but it's not as easy as just repeating spellcheck buit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #73477:
URL: https://github.com/apache/airflow/pull/73477#discussion_r4068696229
##########
devel-common/src/sphinx_exts/airflow_intersphinx.py:
##########
@@ -22,12 +22,54 @@
from typing import TYPE_CHECKING, Any
from provider_yaml_utils import load_package_data
+from sphinx.util.inventory import InventoryFile
if TYPE_CHECKING:
from sphinx.application import Sphinx
AIRFLOW_ROOT_PATH = Path(os.path.abspath(__file__)).parents[3]
GENERATED_PATH = AIRFLOW_ROOT_PATH / "generated"
+INVENTORY_FILENAME = "objects.inv"
+
+
+def _inventory_for(package_name: str, versioned: bool) -> Path:
+ """
+ Prefer an inventory written earlier in this run, fall back to the one fetched from the published docs.
+
+ The html builder writes its inventory at the root of the build directory; a spelling-only build
+ writes one into its own output directory (see ``_dump_inventory_after_spelling_build``, the
+ directory name mirrors ``AirflowDocsBuilder.log_spelling_output_dir``).
+ """
+ build_dir = GENERATED_PATH / "_build" / "docs" / package_name / ("stable" if versioned else "")
+ built_this_run = (
+ build_dir / INVENTORY_FILENAME,
+ build_dir / f"output-spelling-results-{package_name}" / INVENTORY_FILENAME,
+ )
+ downloaded = GENERATED_PATH / "_inventory_cache" / package_name / INVENTORY_FILENAME
+ return next((inventory for inventory in built_this_run if inventory.exists()), downloaded)
+
+
+class _HtmlTargetUris:
+ """Gives inventory entries the URIs the html build would, so the file is an ordinary inventory."""
+
+ @staticmethod
+ def get_target_uri(docname: str, typ: str | None = None) -> str:
+ return f"{docname}.html"
+
+
+def _dump_inventory_after_spelling_build(app: Sphinx, exception: Exception | None) -> None:
+ """
+ Write ``objects.inv`` after a successful spelling build.
+
+ Cross-references are resolved by every builder, so a ``--spellcheck-only`` run of package B fails
Review Comment:
The problem here is that this breaks apart when:
* Package B -> A and A adds new method
* Package A has unrelated "spelling" mistake
In this case (I believe) failed spellcheck build of A will **not** produce new inventory. As far as I know inventory is produced by spellcheck build **only** when it succeeds. But if there is a single spelling mistake in A - there will be no inventory for it (in this case the only way to get inventory A with new method is to run one complete build without checking spelling).
And the issue in this case is that this might lead to a completely misleading errors that will confuse everyone.
Package B will fail because A has no new method (which it has - but inventory for it has never been build). And that escalates quickly if something depends on package B - then you have several indirecations of misleading errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #73477:
URL: https://github.com/apache/airflow/pull/73477#discussion_r4069686823
##########
.github/workflows/ci-image-checks.yml:
##########
@@ -204,6 +204,8 @@ jobs:
python: "${{ inputs.default-python-version }}"
use-uv: ${{ inputs.use-uv }}
make-mnt-writeable-and-cleanup: true
+ # The docs build only needs the CI image; the runner has ~90 GB free without the cleanup.
+ free-up-disk-space: false
Review Comment:
Fair. I checked the 16 docs jobs on main over the last two days and all of them start with 93 GB free, but that is not a guarantee, so I changed it to a threshold: the docs job now passes `min-free-space-gb: 40` and the prepare action runs the cleanup unless `/` already has at least that much free. The docs job needs roughly 25 GB (image tar, loaded image, built docs). Other jobs are unchanged (no threshold, cleanup always runs). 35c7240783.
##########
devel-common/src/docs/build_docs.py:
##########
@@ -693,52 +729,19 @@ def build_docs(
all_spelling_errors.update(package_spelling_errors)
if not one_pass_only:
- # Build documentation for some packages again if it can help them.
- package_build_errors = retry_building_docs_if_needed(
- all_build_errors=all_build_errors,
- all_spelling_errors=all_spelling_errors,
- autobuild=autobuild,
- docs_only=docs_only,
- jobs=jobs,
- verbose=verbose,
- package_build_errors=package_build_errors,
- originally_built_packages=packages_to_build,
- # If spellchecking fails, we need to rebuild all packages first in case some references
- # are broken between packages
- rebuild_all_packages=spellcheck_only,
Review Comment:
The reason the old code had to rebuild everything with html was that spelling builds produced no inventory. That is the part this PR changes: `airflow_intersphinx` now writes `objects.inv` at the end of a spelling build too, and prefers it over the downloaded one for packages built earlier in the run. So the B -> A case resolves the same way it does in a docs build: B fails the first pass on the missing label, the retry rebuilds only B, and B picks up A's fresh inventory.
I reproduced exactly that locally with `--spellcheck-only ftp sftp`: a new `.. _ftp-brand-new-label:` in the ftp docs, referenced with `:ref:` from the sftp docs (sftp is heavier, so it is scheduled first):
```
apache-airflow-providers-ftp : Finished spell-checking successfully
apache-airflow-providers-sftp : Finished spell-checking with errors
providers/sftp/docs/index.rst:157: WARNING: undefined label: 'apache-airflow-providers-ftp:ftp-brand-new-label'
Some packages failed to build due to dependencies. We need another pass.
Packages to rebuild: ['apache-airflow-providers-sftp']
apache-airflow-providers-sftp : Finished spell-checking successfully
No packages to retry. No more passes are needed.
Documentation build is successful
```
The A -> B -> C chain still gets the two retry passes (three passes total); they are no-ops when nothing failed. The retry predicate is the one that was already there (retry only when an error is one of the "missing thing" kinds).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #73477:
URL: https://github.com/apache/airflow/pull/73477#discussion_r4068482667
##########
.github/workflows/ci-image-checks.yml:
##########
@@ -204,6 +204,8 @@ jobs:
python: "${{ inputs.default-python-version }}"
use-uv: ${{ inputs.use-uv }}
make-mnt-writeable-and-cleanup: true
+ # The docs build only needs the CI image; the runner has ~90 GB free without the cleanup.
+ free-up-disk-space: false
Review Comment:
Small thing - this is not **always** true. Sometimes we get the runner with far less of free space.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #73477:
URL: https://github.com/apache/airflow/pull/73477#discussion_r4070126458
##########
devel-common/src/docs/build_docs.py:
##########
@@ -693,52 +729,19 @@ def build_docs(
all_spelling_errors.update(package_spelling_errors)
if not one_pass_only:
- # Build documentation for some packages again if it can help them.
- package_build_errors = retry_building_docs_if_needed(
- all_build_errors=all_build_errors,
- all_spelling_errors=all_spelling_errors,
- autobuild=autobuild,
- docs_only=docs_only,
- jobs=jobs,
- verbose=verbose,
- package_build_errors=package_build_errors,
- originally_built_packages=packages_to_build,
- # If spellchecking fails, we need to rebuild all packages first in case some references
- # are broken between packages
- rebuild_all_packages=spellcheck_only,
Review Comment:
Yes. The inventory is written whenever the build runs to completion, whatever it failed on, because under `-W` Sphinx (8.1 in the CI image, same code in 9.x) counts warnings and sets the exit status at the end instead of aborting on the first one. Only an actual exception part-way through skips it, since the environment may not hold every document then.
Tried what you suggested, same ftp/sftp setup (new label in ftp, referenced from sftp), no misspelling this time:
Case: title underline too short plus unexpected indentation in ftp:
```
apache-airflow-providers-ftp : Finished spell-checking with errors
providers/ftp/docs/index.rst:167: WARNING: Title underline too short.
build finished with problems, 2 warnings (with warnings treated as errors).
apache-airflow-providers-sftp : Finished spell-checking with errors
providers/sftp/docs/index.rst:157: WARNING: undefined label: 'apache-airflow-providers-ftp:ftp-brand-new-label'
Packages to rebuild: ['apache-airflow-providers-sftp']
apache-airflow-providers-sftp : Finished spell-checking successfully
No packages to retry. No more passes are needed.
ftp inventory present: yes
```
Case: the above plus an unknown directive in ftp (an ERROR-level docutils message) plus a misspelling:
```
apache-airflow-providers-ftp : Finished spell-checking with errors
providers/ftp/docs/index.rst:167: WARNING: Title underline too short.
providers/ftp/docs/index.rst:172: ERROR: Unknown directive type "thisdirectivedoesnotexist".
WARNING: Found 1 misspelled words
build finished with problems, 4 warnings (with warnings treated as errors).
apache-airflow-providers-sftp : Finished spell-checking with errors <- undefined label, first pass
Packages to rebuild: ['apache-airflow-providers-sftp']
apache-airflow-providers-sftp : Finished spell-checking successfully
No packages to retry. No more passes are needed.
ftp inventory present: yes
```
In both, the run exits 1 and the summary shows only the genuine ftp errors; the "missing label" in sftp is gone after its retry rather than cascading.
The case that still skips the inventory is a real crash in the ftp build (extension exception, autoapi import error). That one cascades on `main` too: the rebuild-all pass builds ftp with html, the html builder writes `objects.inv` in `finish()`, and a crashed build never gets there, so sftp fails on the label again in passes 2, 3 and 4. Same end state, about an hour later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #73477:
URL: https://github.com/apache/airflow/pull/73477#discussion_r4068627780
##########
devel-common/src/docs/build_docs.py:
##########
@@ -693,52 +729,19 @@ def build_docs(
all_spelling_errors.update(package_spelling_errors)
if not one_pass_only:
- # Build documentation for some packages again if it can help them.
- package_build_errors = retry_building_docs_if_needed(
- all_build_errors=all_build_errors,
- all_spelling_errors=all_spelling_errors,
- autobuild=autobuild,
- docs_only=docs_only,
- jobs=jobs,
- verbose=verbose,
- package_build_errors=package_build_errors,
- originally_built_packages=packages_to_build,
- # If spellchecking fails, we need to rebuild all packages first in case some references
- # are broken between packages
- rebuild_all_packages=spellcheck_only,
Review Comment:
This is what resulted in (infrequent) long spellcheck builds in edge cases,because when it failed once during spellchecking, you had to rebuild it **again** and this time all packages and not "spellcheck_only", because you did not know which package needs rebuilding to re-create inventory used.
Spellcheck builds do **not** recreate inventories. This is the problem. So if you run "spellcheck-only" build and it fails because of missing new method in another package + unrelated spellcheck errors, you need to effectively disable "spellcheck-only" and attempt to build full inventoris of all the packages you build - to avoid misleading errors- otherwise you will never have the inventory created with the new methods and most of the things you will see are those (seemingly) missing methods - while spellcheck or other genuine documentation errors (too short header line for example) that are really causing it will be deeply buried.
At this point you are also really guessing it - because it might also be genuine documentation errors not only inventory issue (for example typo). You really genuinly do not know here. Note - we **only** attempt that "rebuild" when there are errors indicating that **likely** there are new objects that need new inventory (There are few types of error that indicate that this might be the case). Some errors will just not be retried - the "rebuild if needed" checks the errors and will not attempt to rebuild if errors are of a different kind - it will rebuild only if all errors are "missing thing".
That **migh** result in some cases indeed the 4 th build was needed to get the docs green and you need switch from doing spellcheck only - because you need to rebuild inventories.
Imagine the case A -> B -> C (i.e. package C adding new thing, used by B and package A using new thing added in B). An example (realistic and sometimes happening) case would be (but this is extremely rare though) - adding common provider method used by SSH provider that adds new method that is used by SFTP provider. Yes. Happened.
This led to 4 (!) builds needed - one spellcheck only, and 3 doc + spellcheck builds:
1) spellcheck build -> failing (and you guess - because of new method added).
2) doc + spellcheck build of everything to rebuild inventory -> failing A,B if they were built before C . For spelcheck we need to build **all** packages not only the failing one, because spellcheck build does not produce new inventory, it only does spellcheck, so we also need to build C, to get the inventory with new methods.And this needs to be really "all packages" build here in case of spellcheck because we need to have all inventories locally - even if some of the packages failed. For example even if C spellcheck worked in 1) - there is missing inventory for C. So we really need to build **everything** here.
3) repeatebuild of A, B (those two failed. if A is build before B it will fail again. Again that one needs to be "full" build because we need also B inventory to be build.
4) build of A.
We could avoid 4 if 3) is done sequentially and if we build it in the right sequence B->A. But for that we would need to sort the remaining builds topologically.
Another option in this case is to switch such PR that adds several things into series of PRs where you knowingly add things in the right order. But that is kinda superpower to know that this is the case and figure out the right sequence and split the PR in the right way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #73477:
URL: https://github.com/apache/airflow/pull/73477#discussion_r4070097195
##########
devel-common/src/docs/build_docs.py:
##########
@@ -693,52 +729,19 @@ def build_docs(
all_spelling_errors.update(package_spelling_errors)
if not one_pass_only:
- # Build documentation for some packages again if it can help them.
- package_build_errors = retry_building_docs_if_needed(
- all_build_errors=all_build_errors,
- all_spelling_errors=all_spelling_errors,
- autobuild=autobuild,
- docs_only=docs_only,
- jobs=jobs,
- verbose=verbose,
- package_build_errors=package_build_errors,
- originally_built_packages=packages_to_build,
- # If spellchecking fails, we need to rebuild all packages first in case some references
- # are broken between packages
- rebuild_all_packages=spellcheck_only,
Review Comment:
In this case it will not succed, but it was more about the sequence and misleading-nesss of errors it produced.
BTW. Maybe it's alraeady better with latest sphinx - because they improved heavily error reporting since I last time looked at it and we migrated to newer sphinx.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #73477:
URL: https://github.com/apache/airflow/pull/73477#issuecomment-5777498416
Backport here @kaxil @vatsrahul1001 -> will be useful for 3.3.* ;)
https://github.com/apache/airflow/pull/73550
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #73477:
URL: https://github.com/apache/airflow/pull/73477#issuecomment-5774327286
### Backport failed to create: v3-3-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/35711128434'> Run details </a>
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>❌</td>
<td>v3-3-test</td>
<td><a href="https://github.com/apache/airflow/commit/b8be9e54beef9174c95ca37fa5c4d1e0764ab0cb"><img src='https://img.shields.io/badge/Commit-b8be9e5-red' alt='Commit Link'></a></td>
</tr>
</table>
You can attempt to backport this manually by running:
```bash
cherry_picker b8be9e5 v3-3-test
```
This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.
After you have resolved the conflicts, you can continue the backport process by running:
```bash
cherry_picker --continue
```
If you don't have cherry-picker installed, see the [installation guide](https://github.com/apache/airflow/blob/main/dev/README.md#how-to-backport-pr-with-cherry-picker-cli).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #73477:
URL: https://github.com/apache/airflow/pull/73477#discussion_r4068525032
##########
devel-common/src/docs/build_docs.py:
##########
@@ -693,52 +729,19 @@ def build_docs(
all_spelling_errors.update(package_spelling_errors)
if not one_pass_only:
- # Build documentation for some packages again if it can help them.
- package_build_errors = retry_building_docs_if_needed(
- all_build_errors=all_build_errors,
- all_spelling_errors=all_spelling_errors,
- autobuild=autobuild,
- docs_only=docs_only,
- jobs=jobs,
- verbose=verbose,
- package_build_errors=package_build_errors,
- originally_built_packages=packages_to_build,
- # If spellchecking fails, we need to rebuild all packages first in case some references
- # are broken between packages
- rebuild_all_packages=spellcheck_only,
Review Comment:
There are some edge cases here that are not handled.
The problem is that if there is a new object added in package A, and we spellcheck package B and A and we run "spellcheck only", and we build it in B-> A sequence, this build will fail - because B will use inventory of A before it is build and it will fail the build, leaving broken APIs - and then the only way to build it (or so I found in the past was to completely build everything before doing spellchecking.
So I think this shortcut will fail a number of cases where for example we are adding a new common provider object/method and use it in another provider. This is quite common case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #73477:
URL: https://github.com/apache/airflow/pull/73477#discussion_r4068661255
##########
devel-common/src/sphinx_exts/docs_build/docs_builder.py:
##########
@@ -39,6 +45,121 @@
console = Console(force_terminal=True, color_system="standard", width=CONSOLE_WIDTH)
+# Sphinx is run in the current process (one worker builds many packages in a row) so that the fixed
Review Comment:
But risky - because of potential side-effects. But likely worth taking the risk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #73477:
URL: https://github.com/apache/airflow/pull/73477#issuecomment-5771173879
This is quite impressive!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil merged PR #73477:
URL: https://github.com/apache/airflow/pull/73477
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #73477:
URL: https://github.com/apache/airflow/pull/73477#discussion_r4068658768
##########
devel-common/src/sphinx_exts/docs_build/docs_builder.py:
##########
@@ -39,6 +45,121 @@
console = Console(force_terminal=True, color_system="standard", width=CONSOLE_WIDTH)
+# Sphinx is run in the current process (one worker builds many packages in a row) so that the fixed
Review Comment:
cleveer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #73477:
URL: https://github.com/apache/airflow/pull/73477#discussion_r4070071460
##########
devel-common/src/docs/build_docs.py:
##########
@@ -693,52 +729,19 @@ def build_docs(
all_spelling_errors.update(package_spelling_errors)
if not one_pass_only:
- # Build documentation for some packages again if it can help them.
- package_build_errors = retry_building_docs_if_needed(
- all_build_errors=all_build_errors,
- all_spelling_errors=all_spelling_errors,
- autobuild=autobuild,
- docs_only=docs_only,
- jobs=jobs,
- verbose=verbose,
- package_build_errors=package_build_errors,
- originally_built_packages=packages_to_build,
- # If spellchecking fails, we need to rebuild all packages first in case some references
- # are broken between packages
- rebuild_all_packages=spellcheck_only,
Review Comment:
Is the inventory also produced when there is another failure ? The big issue was when spellchecking failed for reasons unrelated to spellchecking - and then inventory for that package was not generated --> and that cascaded to other packages having "missing" things from the original package that failed spellchecking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #73477:
URL: https://github.com/apache/airflow/pull/73477#issuecomment-5769620831
Wow.. looks incredible :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #73477:
URL: https://github.com/apache/airflow/pull/73477#discussion_r4068525032
##########
devel-common/src/docs/build_docs.py:
##########
@@ -693,52 +729,19 @@ def build_docs(
all_spelling_errors.update(package_spelling_errors)
if not one_pass_only:
- # Build documentation for some packages again if it can help them.
- package_build_errors = retry_building_docs_if_needed(
- all_build_errors=all_build_errors,
- all_spelling_errors=all_spelling_errors,
- autobuild=autobuild,
- docs_only=docs_only,
- jobs=jobs,
- verbose=verbose,
- package_build_errors=package_build_errors,
- originally_built_packages=packages_to_build,
- # If spellchecking fails, we need to rebuild all packages first in case some references
- # are broken between packages
- rebuild_all_packages=spellcheck_only,
Review Comment:
There are some edge cases here that are not handled.
The problem is that if there is a new object added in package A, and we spellcheck package B and A and we run "spellcheck only", and we build it in B-> A sequence, this build will fail - because B will use inventory of A before it is build and it will fail the build, leaving broken APIs - and then the only way to build it (or so I found in the past) was to completely build everything before doing spellchecking.
So I think this shortcut will fail a number of cases where for example we are adding a new common provider object/method and use it in another provider. This is quite common case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #73477:
URL: https://github.com/apache/airflow/pull/73477#discussion_r4068666384
##########
devel-common/src/sphinx_exts/docs_build/docs_builder.py:
##########
@@ -39,6 +45,121 @@
console = Console(force_terminal=True, color_system="standard", width=CONSOLE_WIDTH)
+# Sphinx is run in the current process (one worker builds many packages in a row) so that the fixed
+# start-up cost of a build is paid once per worker instead of once per package. For a small provider
+# that fixed cost is ~20s out of ~22s: importing airflow, parsing and validating every provider.yaml,
+# and - the largest part - autoapi's astroid parsing of the airflow modules every provider imports.
+# The astroid parse cache in particular stays warm across the packages a worker builds.
+
+
+def _forget_sphinx_conf_modules() -> None:
+ """
+ Drop the shared Sphinx configuration modules so the next conf.py re-executes them from scratch.
+
+ The per-package ``conf.py`` files do ``from docs.provider_conf import *`` (or import
+ ``docs.utils.conf_constants``) and then mutate the lists they get - ``extensions.append(...)``,
+ ``autoapi_ignore.extend(...)``. Reusing the cached module would leak one package's additions into
+ the next build, so everything under ``devel-common/src/docs`` except the build script is forgotten.
+ """
+ docs_sources_prefix = DOCS_SOURCES_PATH.as_posix()
+ for name, module in list(sys.modules.items()):
+ if name == "docs.build_docs" or not name.startswith("docs."):
+ continue
+ module_file = getattr(module, "__file__", None) or ""
+ if module_file.startswith(docs_sources_prefix):
+ del sys.modules[name]
+
+
+class _RedirectableStream:
Review Comment:
Very clever :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #73477:
URL: https://github.com/apache/airflow/pull/73477#discussion_r4069688993
##########
devel-common/src/sphinx_exts/airflow_intersphinx.py:
##########
@@ -22,12 +22,54 @@
from typing import TYPE_CHECKING, Any
from provider_yaml_utils import load_package_data
+from sphinx.util.inventory import InventoryFile
if TYPE_CHECKING:
from sphinx.application import Sphinx
AIRFLOW_ROOT_PATH = Path(os.path.abspath(__file__)).parents[3]
GENERATED_PATH = AIRFLOW_ROOT_PATH / "generated"
+INVENTORY_FILENAME = "objects.inv"
+
+
+def _inventory_for(package_name: str, versioned: bool) -> Path:
+ """
+ Prefer an inventory written earlier in this run, fall back to the one fetched from the published docs.
+
+ The html builder writes its inventory at the root of the build directory; a spelling-only build
+ writes one into its own output directory (see ``_dump_inventory_after_spelling_build``, the
+ directory name mirrors ``AirflowDocsBuilder.log_spelling_output_dir``).
+ """
+ build_dir = GENERATED_PATH / "_build" / "docs" / package_name / ("stable" if versioned else "")
+ built_this_run = (
+ build_dir / INVENTORY_FILENAME,
+ build_dir / f"output-spelling-results-{package_name}" / INVENTORY_FILENAME,
+ )
+ downloaded = GENERATED_PATH / "_inventory_cache" / package_name / INVENTORY_FILENAME
+ return next((inventory for inventory in built_this_run if inventory.exists()), downloaded)
+
+
+class _HtmlTargetUris:
+ """Gives inventory entries the URIs the html build would, so the file is an ordinary inventory."""
+
+ @staticmethod
+ def get_target_uri(docname: str, typ: str | None = None) -> str:
+ return f"{docname}.html"
+
+
+def _dump_inventory_after_spelling_build(app: Sphinx, exception: Exception | None) -> None:
+ """
+ Write ``objects.inv`` after a successful spelling build.
+
+ Cross-references are resolved by every builder, so a ``--spellcheck-only`` run of package B fails
Review Comment:
I expected the same and tested it before answering, and it does not break: the inventory is written even when A fails on spelling. Under `-W`, Sphinx 8.1 (and 9.x, same code) does not abort on the first warning; it counts warnings and sets a non-zero exit status at the end of the build ("build finished with problems, N warnings (with warnings treated as errors)"), so `build-finished` still fires with `exception=None` after a complete build and the inventory is complete. Only a build that crashed part-way skips the dump.
Same reproduction as above plus a misspelled sentence in the ftp docs:
```
apache-airflow-providers-ftp : Finished spell-checking with errors <- the 3 misspelled words
apache-airflow-providers-sftp : Finished spell-checking with errors <- undefined label, first pass
Packages to rebuild: ['apache-airflow-providers-sftp']
apache-airflow-providers-sftp : Finished spell-checking successfully <- loaded ftp's fresh inventory
No packages to retry. No more passes are needed.
```
and the final summary lists only the three misspellings in ftp, nothing about the label. The sftp log shows which file it loaded: `.../apache-airflow-providers-ftp/stable/output-spelling-results-apache-airflow-providers-ftp/objects.inv`. I added this to the hook's docstring so it does not look like an accident.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #73477:
URL: https://github.com/apache/airflow/pull/73477#discussion_r4070143251
##########
devel-common/src/sphinx_exts/airflow_intersphinx.py:
##########
@@ -22,12 +22,54 @@
from typing import TYPE_CHECKING, Any
from provider_yaml_utils import load_package_data
+from sphinx.util.inventory import InventoryFile
if TYPE_CHECKING:
from sphinx.application import Sphinx
AIRFLOW_ROOT_PATH = Path(os.path.abspath(__file__)).parents[3]
GENERATED_PATH = AIRFLOW_ROOT_PATH / "generated"
+INVENTORY_FILENAME = "objects.inv"
+
+
+def _inventory_for(package_name: str, versioned: bool) -> Path:
+ """
+ Prefer an inventory written earlier in this run, fall back to the one fetched from the published docs.
+
+ The html builder writes its inventory at the root of the build directory; a spelling-only build
+ writes one into its own output directory (see ``_dump_inventory_after_spelling_build``, the
+ directory name mirrors ``AirflowDocsBuilder.log_spelling_output_dir``).
+ """
+ build_dir = GENERATED_PATH / "_build" / "docs" / package_name / ("stable" if versioned else "")
+ built_this_run = (
+ build_dir / INVENTORY_FILENAME,
+ build_dir / f"output-spelling-results-{package_name}" / INVENTORY_FILENAME,
+ )
+ downloaded = GENERATED_PATH / "_inventory_cache" / package_name / INVENTORY_FILENAME
+ return next((inventory for inventory in built_this_run if inventory.exists()), downloaded)
+
+
+class _HtmlTargetUris:
+ """Gives inventory entries the URIs the html build would, so the file is an ordinary inventory."""
+
+ @staticmethod
+ def get_target_uri(docname: str, typ: str | None = None) -> str:
+ return f"{docname}.html"
+
+
+def _dump_inventory_after_spelling_build(app: Sphinx, exception: Exception | None) -> None:
+ """
+ Write ``objects.inv`` after a successful spelling build.
+
+ Cross-references are resolved by every builder, so a ``--spellcheck-only`` run of package B fails
Review Comment:
Ok. that alleviates my concerts. Fantastic job @kaxil !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up documentation build and spellcheck in CI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #73477:
URL: https://github.com/apache/airflow/pull/73477#discussion_r4070156547
##########
.github/workflows/ci-image-checks.yml:
##########
@@ -204,6 +204,8 @@ jobs:
python: "${{ inputs.default-python-version }}"
use-uv: ${{ inputs.use-uv }}
make-mnt-writeable-and-cleanup: true
+ # The docs build only needs the CI image; the runner has ~90 GB free without the cleanup.
+ free-up-disk-space: false
Review Comment:
Very clever. Yes - we should also set min GB elsewhere, that should speed up startup in other jobs!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org