[PR] Add `breeze registry backfill` command for older provider versions [airflow]

["kaxil (via GitHub)" <gi...@apache.org>]

kaxil opened a new pull request, #63269:
URL: https://github.com/apache/airflow/pull/63269

   Adds `breeze registry backfill` — a new breeze subcommand that extracts runtime parameters and connection types for previously released provider versions. Uses `uv run --with` to install the specific version in a temporary environment, so **no Docker or breeze CI image needed**.
   
   ```bash
   # Single version
   breeze registry backfill --provider amazon --version 9.15.0
   
   # Multiple versions
   breeze registry backfill --provider amazon --version 9.15.0 --version 9.14.0
   ```
   
   The command:
   1. Reads `provider.yaml` + `pyproject.toml` to find the package name and optional extras
   2. Patches `dev/registry/providers.json` to the target version (restores on completion)
   3. Runs `extract_parameters.py` and `extract_connections.py` via `uv run --with {pkg}=={ver}`
   4. Falls back to installing without extras if the full spec fails (handles cases like Google's `leveldb` C dependency)
   
   Output goes to `registry/src/_data/versions/{provider}/{version}/parameters.json` and `connections.json`.
   
   ## Design decisions
   
   **Why `uv run --with` instead of Docker?** The extraction scripts just need the provider package installed to discover modules via `issubclass()`. `uv` handles this in seconds vs minutes for Docker image builds. Docker is still used by `breeze registry extract-data` for the full (latest version) extraction.
   
   **Why sequential per-provider?** `extract_parameters.py` and `extract_connections.py` write to shared files (`modules.json`, `providers.json`). Parallel local runs would clobber each other. The new `registry-backfill.yml` workflow handles parallelization via GitHub Actions matrix strategy — each provider runs in its own job.
   
   **Why retry without extras?** Some providers have extras that require native C libraries (e.g., Google's `leveldb` extra requires `plyvel` which needs `leveldb` headers). The fallback ensures extraction still works for the base package.
   
   Working on sequential and retry with extras in a separate PR
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add `breeze registry backfill` command for older provider versions [airflow]

["kaxil (via GitHub)" <gi...@apache.org>]

kaxil merged PR #63269:
URL: https://github.com/apache/airflow/pull/63269


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add `breeze registry backfill` command for older provider versions [airflow]

["kaxil (via GitHub)" <gi...@apache.org>]

kaxil commented on PR #63269:
URL: https://github.com/apache/airflow/pull/63269#issuecomment-4032108603

   > Nice. Just needs to pass tests :)
   
   Looks like all of test failures are unrelated:
   
   ```
     Traceback (most recent call last):
       File "/home/runner/work/airflow/airflow/task-sdk/dev/generate_task_sdk_models.py", line 99, in <module>
         generate_file()
       File "/home/runner/work/airflow/airflow/task-sdk/dev/generate_task_sdk_models.py", line 83, in generate_file
         latest_version = app.app.versions.version_values[0]
                          ^^^^^^^
       File "<attrs generated getattr airflow.api_fastapi.execution_api.app.InProcessExecutionAPI>", line 6, in __getattr__
       File "/home/runner/work/airflow/airflow/airflow-core/src/airflow/api_fastapi/execution_api/app.py", line 317, in app
         from airflow.api_fastapi.execution_api.routes.connections import has_connection_access
       File "/home/runner/work/airflow/airflow/airflow-core/src/airflow/api_fastapi/execution_api/routes/__init__.py", line 22, in <module>
         from airflow.api_fastapi.execution_api.routes import (
       File "/home/runner/work/airflow/airflow/airflow-core/src/airflow/api_fastapi/execution_api/routes/hitl.py", line 32, in <module>
         from airflow.api_fastapi.execution_api.deps import JWTBearerTIPathDep
     ImportError: cannot import name 'JWTBearerTIPathDep' from 'airflow.api_fastapi.execution_api.deps' (/home/runner/work/airflow/airflow/airflow-core/src/airflow/api_fastapi/execution_api/deps.py)
   ```
   
   ```
   Airflow api-server: ....................................................................................................ERROR: Maximum number of retries while checking service. Exiting 
   Service could not be started!
   ```
   
   and
   
   ```
     airflow-core/src/airflow/api_fastapi/execution_api/routes/hitl.py:32: error: Module "airflow.api_fastapi.execution_api.deps" has no attribute "JWTBearerTIPathDep"  [attr-defined]
         from airflow.api_fastapi.execution_api.deps import JWTBearerTIPathDep
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org