You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kaxil (via GitHub)" <gi...@apache.org> on 2026/04/28 15:07:53 UTC

[PR] Make registry-backfill workflow actually publish backfilled pages [airflow]

kaxil opened a new pull request, #66027:
URL: https://github.com/apache/airflow/pull/66027

   ## Summary
   
   The `Registry Backfill` workflow has been reporting success while uploading nothing. Three related bugs combine:
   
   1. **Cached `providers.json` overwrite drops the backfill version.** `registry-backfill.yml`'s `Download data files from S3 for build` step writes the cached S3 `providers.json` to `registry/src/_data/providers.json` AFTER extraction has run. The cached copy predates the backfilled version, so `provider.versions[]` doesn't list it. `registry/src/_data/providerVersions.js:48-50` then intersects `provider.versions ∩ availableSet` and drops the on-disk `metadata.json` for the new version. Eleventy emits no per-version page.
   
   2. **Silent `aws s3 sync` on missing source.** With (1) preventing the page from being built, the next step's `aws s3 sync` exits 0 on the missing source and the job goes green having uploaded nothing.
   
   3. **A redundant `Extract version metadata from git tags` step that was also broken.** It duplicated `breeze registry backfill`'s internal call to `extract_versions.py` AND was broken for multi-version matrix entries because `extract_versions.py:446` declares `--version` as single-valued (no `action="append"`). When the workflow looped `--version A --version B` argparse only kept the last one. The trailing `|| true` masked the failure.
   
   ## Fix
   
   - **New `dev/registry/patch_providers_json.py`** runs after the S3 download and appends the backfilled version(s) into `provider.versions[]`. Defensively keeps `provider.version` (latest) in the array so patching a previously-empty list doesn't leave the latest field excluded once the array becomes authoritative for `providerVersions.js`.
   
   - **Pre-sync guards** assert (a) the per-version `index.html` was emitted, (b) the API directory exists, (c) `parameters.json` is non-empty, before each `aws s3 sync`. Silent no-ops become hard failures with `::error::` annotations.
   
   - **Delete the redundant standalone `Extract version metadata` step.** `breeze registry backfill` already runs `_run_extract_versions` internally and propagates exit codes correctly.
   
   ## Why patch `providers.json` instead of changing the intersection in `providerVersions.js`
   
   The intersection is shared by `registry-build.yml` (full builds) and `registry-backfill.yml`. Changing it affects both flows. Patching is scoped to backfill, mirrors the mental model that `providers.json` IS the source of truth for "which versions to expose", and matches what `breeze registry publish-versions` is supposed to do at the end of the run anyway.
   
   ## Gotchas
   
   - `packaging` is now declared as a runtime dep of `dev/registry/`. It was already available transitively via `pydantic`, but the patch script uses `packaging.version` for newest-first sorting and shouldn't rely on a transitive dep.
   - The pre-sync guard's content checks deliberately don't assert `classes` is non-empty: providers like `common-compat` legitimately have zero modules ("Configuration-only provider"), and the page renders a placeholder for them.
   
   ## Known follow-up (separate PR)
   
   The `publish-versions` job runs in a fresh checkout where `providers.json` is gitignored, so `publish_registry_versions.py:119-127` raises `FileNotFoundError` after the per-version sync. Tracked separately and shipping right after this PR; not in scope here.
   
   ## Verification
   
   This PR alone makes the per-version pages land on S3 correctly. Once the follow-up `publish-versions` fix lands, the smoke test (`amazon/9.24.0 google/21.0.0 common-compat/1.14.2`) should go fully green:
   
   ```
   gh workflow run registry-backfill.yml -R apache/airflow \
     -f destination=staging \
     -f provider-versions="amazon/9.24.0 google/21.0.0 common-compat/1.14.2"
   ```
   
   Tests in `dev/registry/tests/test_patch_providers_json.py` cover 11 cases including the empty-list / latest-field edges.
   
   Linked prior fixes that built the chain to here: #65972, #65975, #65987, #65984, #65989.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make registry-backfill workflow actually publish backfilled pages [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil merged PR #66027:
URL: https://github.com/apache/airflow/pull/66027


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Make registry-backfill workflow actually publish backfilled pages [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #66027:
URL: https://github.com/apache/airflow/pull/66027#issuecomment-4338999622

   ### Backport failed to create: v3-2-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/25077206893'> Run details </a>
   
   Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
   the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
   
   In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
   
   <table>
       <tr>
           <th>Status</th>
           <th>Branch</th>
           <th>Result</th>
       </tr>
       <tr>
           <td>❌</td>
           <td>v3-2-test</td>
           <td><a href="https://github.com/apache/airflow/commit/7e49884005d8d3cbc5692aa6b2b6062958220fe1"><img src='https://img.shields.io/badge/Commit-7e49884-red' alt='Commit Link'></a></td>
       </tr>
   </table>
   
   You can attempt to backport this manually by running:
   
   ```bash
   cherry_picker 7e49884 v3-2-test
   ```
   
   This should apply the commit to the v3-2-test branch and leave the commit in conflict state marking
   the files that need manual conflict resolution.
   
   After you have resolved the conflicts, you can continue the backport process by running:
   
   ```bash
   cherry_picker --continue
   ```
   
   If you don't have cherry-picker installed, see the [installation guide](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#how-to-backport-pr-with-cherry-picker-cli).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org