You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "zozo123 (via GitHub)" <gi...@apache.org> on 2026/10/04 05:57:38 UTC
[PR] Preflight CI image snapshot metadata before download [airflow]
zozo123 opened a new pull request, #74176:
URL: https://github.com/apache/airflow/pull/74176
## Summary
This is a follow-up to #74173. It adds a small compatibility-manifest preflight before downloading the multi-gigabyte Docker data-directory snapshot.
The producer uploads the existing snapshot metadata as a separate, tiny artifact. Consumers download and validate that metadata first (checkout revision, Docker daemon fingerprint, storage driver/root, and empty-daemon safety conditions). The large snapshot is downloaded only after the manifest matches. Any missing artifact, unknown metadata, mismatch, or validation failure follows the existing stash restore/load path.
## Why
A snapshot that cannot be used should not incur the transfer cost of the full archive. This is an acceleration gate, not a correctness shortcut: the complete snapshot validation still runs after download, and the current image-stash path remains the fallback.
## Validation
- `pytest -q scripts/tests/ci/test_docker_data_root_snapshot.py` — 25 passed
- `bash -n scripts/ci/docker_data_root_snapshot.sh`
- `git diff --check`
## Measurement after merge
Please compare snapshot metadata-match rate, avoided large-artifact downloads, fallback rate, snapshot transfer/restore time, and end-to-end image-preparation time against #74173. No cache-hit or time-saving rate is assumed.
## Dependency
This branch is based on #74173 and should be reviewed/merged after that implementation, or rebased when the snapshot producer/consumer changes land.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Preflight CI image snapshot metadata before download [airflow]
Posted by "zozo123 (via GitHub)" <gi...@apache.org>.
zozo123 closed pull request #74176: Preflight CI image snapshot metadata before download
URL: https://github.com/apache/airflow/pull/74176
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Preflight CI image snapshot metadata before download [airflow]
Posted by "zozo123 (via GitHub)" <gi...@apache.org>.
zozo123 commented on PR #74176:
URL: https://github.com/apache/airflow/pull/74176#issuecomment-6084682269
Closing to stay within the open-PR limit: this builds on #74173 and will be reopened once that lands.
---
Drafted-by: Claude Code (Opus 5.5) (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org