You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "zozo123 (via GitHub)" <gi...@apache.org> on 2026/10/02 08:29:43 UTC
[PR] Shrink the CI image by skipping unused Rust docs and the Go archive [airflow]
zozo123 opened a new pull request, #74076:
URL: https://github.com/apache/airflow/pull/74076
Every CI job restores and loads the CI image. The OS-dependencies layer installs Rust with
rustup's default profile, which adds rust-docs, clippy and rustfmt on top of rustc and cargo.
`install_golang` also leaves the downloaded Go archive in the image root. In the published
`ghcr.io/apache/airflow/main/ci/python3.10:latest` (amd64):
| What | Size | Files |
|---|---|---|
| `usr/local/rustup/toolchains/*/share/doc/rust` (rust-docs) | 744 MB | 65,774 |
| clippy and rustfmt binaries | ~26 MB | 4 |
| `/go1.27.1.linux.tar.gz` | 71 MB, already compressed | 1 |
rust-docs alone is 57% of the files in that layer. Nothing in the repository uses rust-docs, clippy
or rustfmt. Building wheels from source only needs rustc and cargo, and `--profile minimal`
installs exactly `rustc`, `rust-std` and `cargo`.
### Checks run
- **The changed functions, in a clean `debian:bookworm-slim`.** rustup installs rustc 1.99.0 and
cargo with components `cargo`, `rust-std` and `rustc`, and a new crate builds and runs. The rustup
directory is 499 MB in 155 files, versus 1,357 MB in about 66k files in the published image.
Go 1.27.1 works, and no archive is left behind.
- **Load time.** I rebuilt the published CI image from its GHCR layers with rust-docs and the Go
archive removed (65,775 files, 814 MB; the layer went 1,048 → 884 MB compressed). Each variant then
went through `docker save | zstd -6`, as the CI stash does, and was loaded 10 times alternating on
the same machine. The slimmer image loaded faster in all 10 rounds, median −1.25 s (−6.6%), and its
stash is 4.9% smaller (2.47 → 2.35 GB). Loads on that machine take 18–19 s against about 100–120 s
on CI runners, so this PR's own CI run is the real measure.
- **Lint and hooks.** hadolint, shellcheck, and prek's pre-commit and manual stages pass.
`update-inlined-dockerfile-scripts` regenerated the inlined copies in `Dockerfile` and
`Dockerfile.ci`.
The PROD image changes only in its build stage; the runtime stage does not install rustup.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 5.5)
Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Shrink the CI image by skipping unused Rust docs and the Go archive [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #74076:
URL: https://github.com/apache/airflow/pull/74076#issuecomment-5966564352
@zozo123 Nice optimization!!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Shrink the CI image by skipping unused Rust docs and the Go archive [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 merged PR #74076:
URL: https://github.com/apache/airflow/pull/74076
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org