You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by sh...@apache.org on 2026/07/03 17:37:00 UTC
(airflow) branch main updated: Auto-generate sorted packages.txt during providers PyPI upload (#69356)
This is an automated email from the ASF dual-hosted git repository.
shahar1 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new bc2628e4f34 Auto-generate sorted packages.txt during providers PyPI upload (#69356)
bc2628e4f34 is described below
commit bc2628e4f3455d76d4db90cb4650d1b55d6e27b3
Author: Shahar Epstein <60...@users.noreply.github.com>
AuthorDate: Fri Jul 3 20:36:52 2026 +0300
Auto-generate sorted packages.txt during providers PyPI upload (#69356)
Release managers previously copied the PyPI URLs printed by twine out of the
terminal by hand, sorted them, and saved them off to the side for the vote
email and announcement. The completeness-check step also referenced
packages.txt before it existed, and the twine upload used dist/* globs that
swept in the ASF source tarball and the .asc/.sha512 files, which are SVN-only
and make twine fail with 'Unknown distribution format'.
The upload now writes the log and a sorted, de-duplicated packages.txt into the
git-ignored files/ directory, uploads only the provider wheels and sdists, and
the completeness gate runs after the file is generated, right before the vote
email is sent.
---
dev/README_RELEASE_PROVIDERS.md | 79 ++++++++++++++++++++++++++---------------
1 file changed, 50 insertions(+), 29 deletions(-)
diff --git a/dev/README_RELEASE_PROVIDERS.md b/dev/README_RELEASE_PROVIDERS.md
index 7a95c062dd1..7f4c1ab0350 100644
--- a/dev/README_RELEASE_PROVIDERS.md
+++ b/dev/README_RELEASE_PROVIDERS.md
@@ -592,20 +592,6 @@ svn commit -m "Add artifacts for Airflow Providers ${RELEASE_DATE}"
cd "$AIRFLOW_REPO_ROOT"
```
-* Before sending the vote email, gate on the same completeness check the PMC verifiers run, so a
- missing artifact (e.g. the `-source.tar.gz` tarball) fails here instead of in the vote thread.
- Put the package list from the upcoming vote email into `dev/packages.txt`, then run:
-
-```shell script
-cd "$AIRFLOW_REPO_ROOT"
-breeze release-management check-release-files providers --release-date "${RELEASE_DATE}" \
- --packages-file ./dev/packages.txt \
- --path-to-airflow-svn "$(cd ../asf-dist/dev/airflow && pwd -P)"
-```
-
- It exits non-zero and lists every missing file (including `.asc`/`.sha512` variants) if anything is
- absent. Only proceed to the vote once it prints `All expected files are present!`.
-
Verify that the files are available in the ${RELEASE_DATE} folder under
[providers](https://dist.apache.org/repos/dist/dev/airflow/providers/)
@@ -692,15 +678,27 @@ twine check ${AIRFLOW_REPO_ROOT}/dist/*
This is a defence-in-depth practice: the RM machine becomes a one-time
release vehicle, not a persistent point of compromise.
-* Upload the package to PyPI:
+* Upload the packages to PyPI. Only the provider wheels and sdists are uploaded — the ASF
+ `-source.tar.gz` tarball and the `.asc`/`.sha512` files are SVN-only and must not go to PyPI.
+ The output is teed to a log and the PyPI URLs twine prints are extracted (sorted, de-duplicated)
+ into `files/packages.txt`. That file is what you paste into the vote email, and what the
+ completeness gate ("Prepare voting email" section below) and the PMC verifiers consume. Both files
+ land in `files/` (git-ignored), so they are never accidentally committed:
```shell script
-twine upload -r pypi ${AIRFLOW_REPO_ROOT}/dist/*
+mkdir -p "${AIRFLOW_REPO_ROOT}/files"
+# COLUMNS=200 stops rich (twine's printer) from wrapping long URLs when stdout is a pipe.
+COLUMNS=200 twine upload -r pypi ${AIRFLOW_REPO_ROOT}/dist/*.whl \
+ $(ls ${AIRFLOW_REPO_ROOT}/dist/*.tar.gz | grep -v -- '-source.tar.gz') 2>&1 \
+ | tee "${AIRFLOW_REPO_ROOT}/files/twine-upload.log"
+# Only trust packages.txt if the upload above succeeded (PIPESTATUS[0] == 0):
+grep -oE 'https://pypi\.org/project/[^[:space:]]+' "${AIRFLOW_REPO_ROOT}/files/twine-upload.log" \
+ | sort -u > "${AIRFLOW_REPO_ROOT}/files/packages.txt"
```
-* Confirm that the packages are available under the links printed and look good.
-
-* Save these links for later, you'll need to paste them in the email you'll send to dev@airflow.apache.org
+* Confirm that the packages are available under the links printed and look good. The same links are
+ now saved, sorted, in `files/packages.txt` — you'll paste them into the vote email you send to
+ dev@airflow.apache.org.
## Push the RC tags
@@ -856,6 +854,21 @@ gh issue create --repo apache/airflow \
Make sure the packages are in https://dist.apache.org/repos/dist/dev/airflow/providers/
+* Before sending the vote email, gate on the same completeness check the PMC verifiers run, so a
+ missing artifact (e.g. the `-source.tar.gz` tarball) fails here instead of in the vote thread.
+ `files/packages.txt` was already generated by the PyPI upload step ("Publish the Regular
+ distributions to PyPI" above); run the check against it:
+
+```shell script
+cd "$AIRFLOW_REPO_ROOT"
+breeze release-management check-release-files providers --release-date "${RELEASE_DATE}" \
+ --packages-file ./files/packages.txt \
+ --path-to-airflow-svn "$(cd ../asf-dist/dev/airflow && pwd -P)"
+```
+
+ It exits non-zero and lists every missing file (including `.asc`/`.sha512` variants) if anything is
+ absent. Only proceed to the vote once it prints `All expected files are present!`.
+
Send out a vote to the dev@airflow.apache.org mailing list. Here you can prepare text of the
email.
@@ -1004,13 +1017,14 @@ to verify that all expected files are present in SVN. This command will produce
may help with verifying installation of the packages.
Once you have cloned/updated the SVN repository, copy the PyPi URLs shared
-in the email to a file called `packages.txt` in the $AIRFLOW_REPO_ROOT/dev
-directory.
+in the email to a file called `packages.txt` in the $AIRFLOW_REPO_ROOT/files
+directory (git-ignored, so it won't be accidentally committed).
```shell script
cd "$AIRFLOW_REPO_ROOT"
+mkdir -p files
# Copy packages.txt extracted from the mail sent by the release manager here
-breeze release-management check-release-files providers --release-date "${RELEASE_DATE}" --packages-file ./dev/packages.txt --path-to-airflow-svn "${PATH_TO_AIRFLOW_SVN}"
+breeze release-management check-release-files providers --release-date "${RELEASE_DATE}" --packages-file ./files/packages.txt --path-to-airflow-svn "${PATH_TO_AIRFLOW_SVN}"
```
After the above command completes you can build `Dockerfile.pmc` to trigger an installation of each provider
@@ -1566,17 +1580,24 @@ twine check ${AIRFLOW_REPO_ROOT}/dist/*.whl ${AIRFLOW_REPO_ROOT}/dist/*.tar.gz
This is a defence-in-depth practice: the RM machine becomes a one-time
release vehicle, not a persistent point of compromise.
-* Upload the package to PyPI:
+* Upload the packages to PyPI. Only the provider wheels and sdists are uploaded — the ASF
+ `-source.tar.gz` tarball and the `.asc`/`.sha512` files are SVN-only. The PyPI URLs twine prints
+ are extracted (sorted, de-duplicated) into `files/packages.txt` for the announcement message; both
+ files land in `files/` (git-ignored):
```shell script
-twine upload -r pypi ${AIRFLOW_REPO_ROOT}/dist/*.whl ${AIRFLOW_REPO_ROOT}/dist/*.tar.gz
+mkdir -p "${AIRFLOW_REPO_ROOT}/files"
+# COLUMNS=200 stops rich (twine's printer) from wrapping long URLs when stdout is a pipe.
+COLUMNS=200 twine upload -r pypi ${AIRFLOW_REPO_ROOT}/dist/*.whl \
+ $(ls ${AIRFLOW_REPO_ROOT}/dist/*.tar.gz | grep -v -- '-source.tar.gz') 2>&1 \
+ | tee "${AIRFLOW_REPO_ROOT}/files/twine-upload.log"
+# Only trust packages.txt if the upload above succeeded (PIPESTATUS[0] == 0):
+grep -oE 'https://pypi\.org/project/[^[:space:]]+' "${AIRFLOW_REPO_ROOT}/files/twine-upload.log" \
+ | sort -u > "${AIRFLOW_REPO_ROOT}/files/packages.txt"
```
-* Verify that the packages are available under the links printed.
-
-Copy links to updated packages, sort it alphabetically and save it on the side. You will need it for the announcement message.
-
-* Again, confirm that the packages are available under the links printed.
+* Confirm that the packages are available under the links printed. The sorted list is saved in
+ `files/packages.txt` for the announcement message.
## Add the final release tag in git