You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2026/06/04 01:30:09 UTC
[PR] Helm release docs: keep only latest in dist/release (ASF policy) [airflow]
potiuk opened a new pull request, #67988:
URL: https://github.com/apache/airflow/pull/67988
The "Remove old releases" step in the Helm chart release README kept the last
2 chart versions and shipped a broken command: `cd cd`, a working-copy path
that does not match where `asf-dist` is actually checked out, and a single
hardcoded `PREVIOUS_VERSION`.
ASF release policy requires `dist/release` to hold only the latest release,
with superseded versions served from `archive.apache.org`. This updates the
step to:
- remove every version except the one just released,
- fix the working-copy path to reuse `${AIRFLOW_SVN_RELEASE_HELM}` (set in the
publish step) and drop the `cd cd` typo,
- explain why removal must wait until the new `index.yaml` is published (so
users who have not run `helm repo update` are not left with 404ing URLs).
Docs-only change to `dev/README_RELEASE_HELM_CHART.md`.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 4.8)
Generated-by: Claude Code (Opus 4.8) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Helm release docs: keep only latest in dist/release (ASF policy) [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #67988:
URL: https://github.com/apache/airflow/pull/67988
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Helm release docs: keep only latest in dist/release (ASF policy) [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #67988:
URL: https://github.com/apache/airflow/pull/67988#discussion_r3358465148
##########
dev/README_RELEASE_HELM_CHART.md:
##########
@@ -1169,17 +1172,27 @@ Do not add `-dev` suffix to the version.
## Remove old releases
-We should keep the old version a little longer than a day or at least until the updated
-``index.yaml`` is published. This is to avoid errors for users who haven't run ``helm repo update``.
+ASF release policy requires that the distribution area (``dist/release``) holds only the
+latest release; every superseded version must be removed from ``dist/release`` once a new
+version is published. The removed releases are not lost -- they remain permanently available
+from the archive at https://archive.apache.org/dist/airflow/helm-chart/. See
+http://www.apache.org/legal/release-policy.html#when-to-archive
-It is probably ok if we leave last 2 versions on release svn repo too.
+Do this only *after* the updated ``index.yaml`` has been published (see the step above), so
+that users who have not yet run ``helm repo update`` are not left pointing at ``dist/release``
+URLs that now 404 -- it is the published ``index.yaml`` that rewrites old-version URLs to the
+archive.
+
+The snippet below removes every version except the one you just released (``${VERSION}``):
```shell
# http://www.apache.org/legal/release-policy.html#when-to-archive
-cd cd ${AIRFLOW_REPO_ROOT}/asf-dist/release/airflow/helm-chart
-export PREVIOUS_VERSION=1.0.0
-svn rm ${PREVIOUS_VERSION}
-svn commit -m "Remove old Helm Chart release: ${PREVIOUS_VERSION}"
+cd "${AIRFLOW_SVN_RELEASE_HELM}"
+svn update
+for old_version in $(svn ls | sed 's:/$::' | grep -vx "${VERSION}"); do
+ svn rm "${old_version}"
+done
+svn commit -m "Remove superseded Helm Chart releases, keeping only ${VERSION} in dist/release per ASF release policy (older releases remain available from https://archive.apache.org/dist/airflow/helm-chart/)"
Review Comment:
Done — simplified to `svn commit -m "Remove superseded Helm Chart releases"` as suggested.
---
Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting
##########
dev/README_RELEASE_HELM_CHART.md:
##########
@@ -1169,17 +1172,27 @@ Do not add `-dev` suffix to the version.
## Remove old releases
-We should keep the old version a little longer than a day or at least until the updated
-``index.yaml`` is published. This is to avoid errors for users who haven't run ``helm repo update``.
+ASF release policy requires that the distribution area (``dist/release``) holds only the
+latest release; every superseded version must be removed from ``dist/release`` once a new
+version is published. The removed releases are not lost -- they remain permanently available
+from the archive at https://archive.apache.org/dist/airflow/helm-chart/. See
+http://www.apache.org/legal/release-policy.html#when-to-archive
-It is probably ok if we leave last 2 versions on release svn repo too.
+Do this only *after* the updated ``index.yaml`` has been published (see the step above), so
+that users who have not yet run ``helm repo update`` are not left pointing at ``dist/release``
+URLs that now 404 -- it is the published ``index.yaml`` that rewrites old-version URLs to the
+archive.
+
Review Comment:
Agreed for the removal — reworked the paragraph to say removal needs no waiting (superseded versions are permanently on archive). I moved the *one* timing concern that does remain to where it actually applies: the **new** version's archive URL. Archive lags the release, so the "Update index.yaml" step now blocks until the new version is live on archive before the index references it. That keeps your "don't worry about index.yaml propagation" property intact — a stale index only ever contains archive URLs, which never 404.
---
Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting
##########
dev/README_RELEASE_HELM_CHART.md:
##########
@@ -993,7 +993,10 @@ git clone https://github.com/apache/airflow-site.git airflow-site
cd airflow-site
curl https://dist.apache.org/repos/dist/dev/airflow/helm-chart/${VERSION}${VERSION_SUFFIX}/index.yaml -o index.yaml
cp ${AIRFLOW_SVN_RELEASE_HELM}/${VERSION}/airflow-${VERSION}.tgz .
-helm repo index --merge ./index.yaml . --url "https://downloads.apache.org/airflow/helm-chart/${VERSION}"
+# Use archive.apache.org (not downloads.apache.org): downloads only keeps the latest
+# releases, while archive permanently retains every released version even after it is
+# removed from downloads.
+helm repo index --merge ./index.yaml . --url "https://archive.apache.org/dist/airflow/helm-chart/${VERSION}"
Review Comment:
You're right — there is. [ASF infra docs](https://infra.apache.org/release-distribution.html) put it at "about a day" worst case (in practice usually minutes, at most a few hours). It only affects the **just-released** version — superseded versions have been on archive since their own release. So rather than point the new version at `downloads` (which reintroduces the Fastly-stale-cache problem we just hit), the step now blocks until the new `.tgz` is actually downloadable from archive before publishing the index entry for it. That way every URL in any served index — fresh or Fastly-stale — always resolves.
---
Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Helm release docs: keep only latest in dist/release (ASF policy) [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #67988:
URL: https://github.com/apache/airflow/pull/67988#issuecomment-4627243666
### Backport failed to create: v3-2-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/26988918455'> Run details </a>
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>❌</td>
<td>v3-2-test</td>
<td><a href="https://github.com/apache/airflow/commit/034649f1d4fdc1724abdec7f86220aa21cc279a9"><img src='https://img.shields.io/badge/Commit-034649f-red' alt='Commit Link'></a></td>
</tr>
</table>
You can attempt to backport this manually by running:
```bash
cherry_picker 034649f v3-2-test
```
This should apply the commit to the v3-2-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.
After you have resolved the conflicts, you can continue the backport process by running:
```bash
cherry_picker --continue
```
If you don't have cherry-picker installed, see the [installation guide](https://github.com/apache/airflow/blob/main/dev/README.md#how-to-backport-pr-with-cherry-picker-cli).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Helm release docs: keep only latest in dist/release (ASF policy) [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #67988:
URL: https://github.com/apache/airflow/pull/67988#discussion_r3359218253
##########
dev/README_RELEASE_HELM_CHART.md:
##########
@@ -986,14 +986,45 @@ The `index.yaml` file on the Airflow website allows users to easily add the Apac
helm repo add apache-airflow https://airflow.apache.org
```
+The index points **every** version -- including the one just released -- at
+`archive.apache.org`, never `downloads.apache.org`. `downloads.apache.org` keeps only the
+latest release and is fronted by a CDN (Fastly) whose cache can serve a stale `200` that turns
+into a real `404` once a version is removed -- exactly the failure that strands users on
+removed-version URLs. `archive.apache.org` is append-only: once a version lands there it is
+never removed, so those URLs resolve forever, no matter how stale a cached copy of `index.yaml`
+is.
+
+This matters because `index.yaml` is itself served from `airflow.apache.org`, which is *also*
+behind Fastly, so a freshly published index can be served stale for the cache TTL. That is
+harmless **as long as every URL the index has ever contained stays resolvable** -- which the
+archive-only policy guarantees for every superseded version. The one case it does not cover
+automatically is the **just-released** version: `archive.apache.org` lags `downloads.apache.org`
+(the archive sync is [documented](https://infra.apache.org/release-distribution.html) as
+happening "about a day" after a release first appears on downloads -- but that is the worst
+case; in practice the sync is usually within minutes, at most a few hours), so its archive URL
+can 404 during that window.
+**Do not publish the index entry for the new version until that version is live on archive** --
+the steps below block until it is. (Removing superseded versions from `dist/release`, by
+contrast, needs no waiting -- they have been on archive since their own release.)
+
To update `index.yaml` for the newly released chart version, follow these steps:
```shell
git clone https://github.com/apache/airflow-site.git airflow-site
cd airflow-site
+
+# archive.apache.org lags the release (worst case ~1 day; usually minutes, at most hours). Block until the
+# new version is actually downloadable from archive before referencing it in the index --
+# otherwise the index could advertise an archive URL that 404s until the sync catches up.
+until curl -fsI "https://archive.apache.org/dist/airflow/helm-chart/${VERSION}/airflow-${VERSION}.tgz" >/dev/null; do
+ echo "airflow-${VERSION}.tgz not on archive.apache.org yet; waiting 10 min..."
+ sleep 600
+done
Review Comment:
It might be nice to have a positive message when it does succeed, otherwise you come back and see it exited, and if you are lucky you show the exit code in your prompt to know it succeeded vs failed somehow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Helm release docs: keep only latest in dist/release (ASF policy) [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #67988:
URL: https://github.com/apache/airflow/pull/67988#discussion_r3359726093
##########
dev/README_RELEASE_HELM_CHART.md:
##########
@@ -986,14 +986,45 @@ The `index.yaml` file on the Airflow website allows users to easily add the Apac
helm repo add apache-airflow https://airflow.apache.org
```
+The index points **every** version -- including the one just released -- at
+`archive.apache.org`, never `downloads.apache.org`. `downloads.apache.org` keeps only the
+latest release and is fronted by a CDN (Fastly) whose cache can serve a stale `200` that turns
+into a real `404` once a version is removed -- exactly the failure that strands users on
+removed-version URLs. `archive.apache.org` is append-only: once a version lands there it is
+never removed, so those URLs resolve forever, no matter how stale a cached copy of `index.yaml`
+is.
+
+This matters because `index.yaml` is itself served from `airflow.apache.org`, which is *also*
+behind Fastly, so a freshly published index can be served stale for the cache TTL. That is
+harmless **as long as every URL the index has ever contained stays resolvable** -- which the
+archive-only policy guarantees for every superseded version. The one case it does not cover
+automatically is the **just-released** version: `archive.apache.org` lags `downloads.apache.org`
+(the archive sync is [documented](https://infra.apache.org/release-distribution.html) as
+happening "about a day" after a release first appears on downloads -- but that is the worst
+case; in practice the sync is usually within minutes, at most a few hours), so its archive URL
+can 404 during that window.
+**Do not publish the index entry for the new version until that version is live on archive** --
+the steps below block until it is. (Removing superseded versions from `dist/release`, by
+contrast, needs no waiting -- they have been on archive since their own release.)
+
To update `index.yaml` for the newly released chart version, follow these steps:
```shell
git clone https://github.com/apache/airflow-site.git airflow-site
cd airflow-site
+
+# archive.apache.org lags the release (worst case ~1 day; usually minutes, at most hours). Block until the
+# new version is actually downloadable from archive before referencing it in the index --
+# otherwise the index could advertise an archive URL that 404s until the sync catches up.
+until curl -fsI "https://archive.apache.org/dist/airflow/helm-chart/${VERSION}/airflow-${VERSION}.tgz" >/dev/null; do
+ echo "airflow-${VERSION}.tgz not on archive.apache.org yet; waiting 10 min..."
+ sleep 600
+done
Review Comment:
Good call — added a success line right after the loop:
```shell
echo "airflow-${VERSION}.tgz is live on archive.apache.org -- proceeding to publish the index."
```
so a successful wait prints a clear confirmation instead of exiting silently. Pushed in `3995822`.
---
Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Helm release docs: keep only latest in dist/release (ASF policy) [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #67988:
URL: https://github.com/apache/airflow/pull/67988#issuecomment-4623058886
> @potiuk I think this may have removed all the previous versions of the helm charts from the chart repository. https://downloads.apache.org/airflow/helm-chart/ The chart index needs to be updated to point to the archive when a new version is released so that users are not directly forced to get a new chart version.
Yes. That's what we usually do. It turned out to be a Fastly Cache purge issue. But I am also changing it now to always use archive links not download. Download should only ever contain latest release (so removal was good) it was really the index problem that should always point to archive and it should make it resilient to those kind of issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Helm release docs: keep only latest in dist/release (ASF policy) [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #67988:
URL: https://github.com/apache/airflow/pull/67988#discussion_r3357126456
##########
dev/README_RELEASE_HELM_CHART.md:
##########
@@ -993,7 +993,10 @@ git clone https://github.com/apache/airflow-site.git airflow-site
cd airflow-site
curl https://dist.apache.org/repos/dist/dev/airflow/helm-chart/${VERSION}${VERSION_SUFFIX}/index.yaml -o index.yaml
cp ${AIRFLOW_SVN_RELEASE_HELM}/${VERSION}/airflow-${VERSION}.tgz .
-helm repo index --merge ./index.yaml . --url "https://downloads.apache.org/airflow/helm-chart/${VERSION}"
+# Use archive.apache.org (not downloads.apache.org): downloads only keeps the latest
+# releases, while archive permanently retains every released version even after it is
+# removed from downloads.
+helm repo index --merge ./index.yaml . --url "https://archive.apache.org/dist/airflow/helm-chart/${VERSION}"
Review Comment:
Is there a delay for the release to be active on archive? For some reason I have a trace of a memory there was and I tried this exact change long ago...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Helm release docs: keep only latest in dist/release (ASF policy) [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #67988:
URL: https://github.com/apache/airflow/pull/67988#discussion_r3356827388
##########
dev/README_RELEASE_HELM_CHART.md:
##########
@@ -1169,17 +1172,27 @@ Do not add `-dev` suffix to the version.
## Remove old releases
-We should keep the old version a little longer than a day or at least until the updated
-``index.yaml`` is published. This is to avoid errors for users who haven't run ``helm repo update``.
+ASF release policy requires that the distribution area (``dist/release``) holds only the
+latest release; every superseded version must be removed from ``dist/release`` once a new
+version is published. The removed releases are not lost -- they remain permanently available
+from the archive at https://archive.apache.org/dist/airflow/helm-chart/. See
+http://www.apache.org/legal/release-policy.html#when-to-archive
-It is probably ok if we leave last 2 versions on release svn repo too.
+Do this only *after* the updated ``index.yaml`` has been published (see the step above), so
+that users who have not yet run ``helm repo update`` are not left pointing at ``dist/release``
+URLs that now 404 -- it is the published ``index.yaml`` that rewrites old-version URLs to the
+archive.
+
Review Comment:
```suggestion
```
Everyone uses archive now, we don't need to worry about timing of index.yaml at all.
##########
dev/README_RELEASE_HELM_CHART.md:
##########
@@ -1169,17 +1172,27 @@ Do not add `-dev` suffix to the version.
## Remove old releases
-We should keep the old version a little longer than a day or at least until the updated
-``index.yaml`` is published. This is to avoid errors for users who haven't run ``helm repo update``.
+ASF release policy requires that the distribution area (``dist/release``) holds only the
+latest release; every superseded version must be removed from ``dist/release`` once a new
+version is published. The removed releases are not lost -- they remain permanently available
+from the archive at https://archive.apache.org/dist/airflow/helm-chart/. See
+http://www.apache.org/legal/release-policy.html#when-to-archive
-It is probably ok if we leave last 2 versions on release svn repo too.
+Do this only *after* the updated ``index.yaml`` has been published (see the step above), so
+that users who have not yet run ``helm repo update`` are not left pointing at ``dist/release``
+URLs that now 404 -- it is the published ``index.yaml`` that rewrites old-version URLs to the
+archive.
+
+The snippet below removes every version except the one you just released (``${VERSION}``):
```shell
# http://www.apache.org/legal/release-policy.html#when-to-archive
-cd cd ${AIRFLOW_REPO_ROOT}/asf-dist/release/airflow/helm-chart
-export PREVIOUS_VERSION=1.0.0
-svn rm ${PREVIOUS_VERSION}
-svn commit -m "Remove old Helm Chart release: ${PREVIOUS_VERSION}"
+cd "${AIRFLOW_SVN_RELEASE_HELM}"
+svn update
+for old_version in $(svn ls | sed 's:/$::' | grep -vx "${VERSION}"); do
+ svn rm "${old_version}"
+done
+svn commit -m "Remove superseded Helm Chart releases, keeping only ${VERSION} in dist/release per ASF release policy (older releases remain available from https://archive.apache.org/dist/airflow/helm-chart/)"
Review Comment:
We don't need this detail in every commit message.
```suggestion
svn commit -m "Remove superseded Helm Chart releases"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Helm release docs: keep only latest in dist/release (ASF policy) [airflow]
Posted by "gschuurman (via GitHub)" <gi...@apache.org>.
gschuurman commented on PR #67988:
URL: https://github.com/apache/airflow/pull/67988#issuecomment-4620546483
@potiuk I think this may have removed all the previous versions of the helm charts from the chart repository. https://downloads.apache.org/airflow/helm-chart/ The chart index needs to be updated to point to the archive when a new version is released so that users are not directly forced to get a new chart version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Helm release docs: keep only latest in dist/release (ASF policy) [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #67988:
URL: https://github.com/apache/airflow/pull/67988#issuecomment-4625559416
All comments addressed I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org