You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2025/01/09 20:45:22 UTC
(airflow) branch apply-workflow-run-removal created (now 6dd1989906d)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch apply-workflow-run-removal
in repository https://gitbox.apache.org/repos/asf/airflow.git
at 6dd1989906d [v2-10-test] Backport pull_request_target removal
This branch includes the following new commits:
new 6dd1989906d [v2-10-test] Backport pull_request_target removal
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(airflow) 01/01: [v2-10-test] Backport pull_request_target removal
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch apply-workflow-run-removal
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6dd1989906dd048b9f87ab16465cb07d545300b0
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Thu Jan 9 21:37:35 2025 +0100
[v2-10-test] Backport pull_request_target removal
This is a bulk change that synchronizes dev/ci scripts for v2-10-test
branch with main #45266 - including follow-ups.
Rather than cherry-picking relevant PRs, this one gets the
latest version of the scripts from main and updates the branch with
some changes to adapt them to v2-10-test (such as bringing back
python 3.8 support, removing some providers checks after the
bulk move of providers and making sure all tests are passing.
This is far easier than cherry-picking the changes, because for
the v2-10-test we stopped cherry-picking CI changes which was
deemed unnecessary (we used to do it for all previous branches)
but this made it far more difficult (if not impossible) to
cherry-pick individual changes.
Fortunately, the CI scripts are maintained in the way that their
latest version **should** in principle work for a v2-* branch and
hopefully after just a few adjustments we should be able to
synchronize the changes from main by updating all relevant
CI/DEV scripts, dockerfile images, workflows, pre-commits etc.
---
.dockerignore | 7 +
.github/actions/breeze/action.yml | 12 +-
.github/actions/checkout_target_commit/action.yml | 81 --
.github/actions/install-pre-commit/action.yml | 58 +-
.github/actions/migration_tests/action.yml | 14 +
.github/actions/post_tests_success/action.yml | 4 +-
.github/actions/prepare_all_ci_images/action.yml | 68 +
.../actions/prepare_breeze_and_image/action.yml | 47 +-
.github/actions/prepare_single_ci_image/action.yml | 50 +
.github/workflows/additional-ci-image-checks.yml | 31 +-
.github/workflows/additional-prod-image-tests.yml | 65 +-
.github/workflows/automatic-backport.yml | 77 ++
.github/workflows/backport-cli.yml | 122 ++
.github/workflows/basic-tests.yml | 221 ++-
.github/workflows/build-images.yml | 259 ----
.github/workflows/ci-image-build.yml | 121 +-
...ic-checks-mypy-docs.yml => ci-image-checks.yml} | 193 ++-
.github/workflows/ci.yml | 467 +++----
.github/workflows/codeql-analysis.yml | 18 +-
.github/workflows/finalize-tests.yml | 30 +-
.github/workflows/generate-constraints.yml | 43 +-
.github/workflows/helm-tests.yml | 23 +-
.github/workflows/integration-system-tests.yml | 207 +++
.github/workflows/integration-tests.yml | 103 --
.github/workflows/k8s-tests.yml | 91 +-
.github/workflows/news-fragment.yml | 81 ++
.github/workflows/prod-image-build.yml | 151 +--
.github/workflows/prod-image-extra-checks.yml | 15 +-
.github/workflows/push-image-cache.yml | 90 +-
.github/workflows/recheck-old-bug-report.yml | 1 +
.github/workflows/release_dockerhub_image.yml | 67 +-
.github/workflows/run-unit-tests.yml | 98 +-
.github/workflows/special-tests.yml | 170 +--
.github/workflows/task-sdk-tests.yml | 93 ++
...ck-providers.yml => test-provider-packages.yml} | 187 ++-
.gitignore | 17 +-
.pre-commit-config.yaml | 301 ++---
Dockerfile | 118 +-
Dockerfile.ci | 229 +---
contributing-docs/08_static_code_checks.rst | 527 ++++----
contributing-docs/testing/integration_tests.rst | 68 +-
dev/breeze/README.md | 88 +-
dev/breeze/doc/01_installation.rst | 72 +-
dev/breeze/doc/02_customizing.rst | 34 +
dev/breeze/doc/03_developer_tasks.rst | 28 +-
dev/breeze/doc/04_troubleshooting.rst | 47 +-
dev/breeze/doc/05_test_commands.rst | 207 +--
dev/breeze/doc/06_managing_docker_images.rst | 113 +-
dev/breeze/doc/09_release_management_tasks.rst | 77 +-
dev/breeze/doc/10_advanced_breeze_topics.rst | 15 +-
.../0002-implement-standalone-python-command.md | 2 +-
.../doc/adr/0016-use-uv-tool-to-install-breeze.md | 56 +
dev/breeze/doc/ci/01_ci_environment.md | 97 +-
dev/breeze/doc/ci/02_images.md | 206 ++-
dev/breeze/doc/ci/03_github_variables.md | 2 +-
dev/breeze/doc/ci/04_selective_checks.md | 178 +--
dev/breeze/doc/ci/05_workflows.md | 241 ++--
dev/breeze/doc/ci/06_debugging.md | 64 +
dev/breeze/doc/ci/06_diagrams.md | 466 -------
dev/breeze/doc/ci/07_debugging.md | 88 --
dev/breeze/doc/ci/07_running_ci_locally.md | 187 +++
dev/breeze/doc/ci/08_running_ci_locally.md | 141 --
dev/breeze/doc/ci/README.md | 5 +-
dev/breeze/doc/images/image_artifacts.png | Bin 0 -> 47666 bytes
dev/breeze/doc/images/output-commands.svg | 104 +-
dev/breeze/doc/images/output_build-docs.svg | 110 +-
dev/breeze/doc/images/output_build-docs.txt | 2 +-
dev/breeze/doc/images/output_ci-image.svg | 34 +-
dev/breeze/doc/images/output_ci-image.txt | 2 +-
dev/breeze/doc/images/output_ci-image_build.svg | 176 ++-
dev/breeze/doc/images/output_ci-image_build.txt | 2 +-
.../images/output_ci-image_export-mount-cache.svg | 118 ++
.../images/output_ci-image_export-mount-cache.txt | 1 +
.../images/output_ci-image_import-mount-cache.svg | 118 ++
.../images/output_ci-image_import-mount-cache.txt | 1 +
dev/breeze/doc/images/output_ci-image_load.svg | 182 +++
dev/breeze/doc/images/output_ci-image_load.txt | 1 +
dev/breeze/doc/images/output_ci-image_pull.svg | 70 +-
dev/breeze/doc/images/output_ci-image_pull.txt | 2 +-
dev/breeze/doc/images/output_ci-image_save.svg | 132 ++
dev/breeze/doc/images/output_ci-image_save.txt | 1 +
dev/breeze/doc/images/output_ci-image_verify.svg | 56 +-
dev/breeze/doc/images/output_ci-image_verify.txt | 2 +-
dev/breeze/doc/images/output_ci.svg | 2 +-
.../output_ci_find-backtracking-candidates.svg | 2 +-
dev/breeze/doc/images/output_ci_fix-ownership.svg | 8 +-
dev/breeze/doc/images/output_ci_free-space.svg | 8 +-
.../doc/images/output_ci_get-workflow-info.svg | 6 +-
dev/breeze/doc/images/output_ci_resource-check.svg | 6 +-
dev/breeze/doc/images/output_cleanup.svg | 10 +-
dev/breeze/doc/images/output_compile-ui-assets.svg | 119 ++
dev/breeze/doc/images/output_compile-ui-assets.txt | 1 +
.../doc/images/output_compile-www-assets.svg | 10 +-
dev/breeze/doc/images/output_down.svg | 14 +-
dev/breeze/doc/images/output_exec.svg | 6 +-
.../doc/images/output_generate-migration-file.svg | 14 +-
dev/breeze/doc/images/output_k8s.svg | 14 +-
.../doc/images/output_k8s_build-k8s-image.svg | 54 +-
.../doc/images/output_k8s_build-k8s-image.txt | 2 +-
.../doc/images/output_k8s_configure-cluster.svg | 6 +-
.../doc/images/output_k8s_configure-cluster.txt | 2 +-
.../doc/images/output_k8s_create-cluster.svg | 6 +-
.../doc/images/output_k8s_create-cluster.txt | 2 +-
.../doc/images/output_k8s_delete-cluster.svg | 4 +-
.../doc/images/output_k8s_delete-cluster.txt | 2 +-
.../doc/images/output_k8s_deploy-airflow.svg | 6 +-
.../doc/images/output_k8s_deploy-airflow.txt | 2 +-
dev/breeze/doc/images/output_k8s_k9s.svg | 4 +-
dev/breeze/doc/images/output_k8s_k9s.txt | 2 +-
dev/breeze/doc/images/output_k8s_logs.svg | 4 +-
dev/breeze/doc/images/output_k8s_logs.txt | 2 +-
.../doc/images/output_k8s_run-complete-tests.svg | 84 +-
.../doc/images/output_k8s_run-complete-tests.txt | 2 +-
dev/breeze/doc/images/output_k8s_setup-env.svg | 8 +-
dev/breeze/doc/images/output_k8s_shell.svg | 4 +-
dev/breeze/doc/images/output_k8s_shell.txt | 2 +-
dev/breeze/doc/images/output_k8s_status.svg | 4 +-
dev/breeze/doc/images/output_k8s_status.txt | 2 +-
dev/breeze/doc/images/output_k8s_tests.svg | 6 +-
dev/breeze/doc/images/output_k8s_tests.txt | 2 +-
.../doc/images/output_k8s_upload-k8s-image.svg | 6 +-
.../doc/images/output_k8s_upload-k8s-image.txt | 2 +-
dev/breeze/doc/images/output_prod-image.svg | 16 +-
dev/breeze/doc/images/output_prod-image.txt | 2 +-
dev/breeze/doc/images/output_prod-image_build.svg | 198 ++-
dev/breeze/doc/images/output_prod-image_build.txt | 2 +-
dev/breeze/doc/images/output_prod-image_load.svg | 170 +++
dev/breeze/doc/images/output_prod-image_load.txt | 1 +
dev/breeze/doc/images/output_prod-image_pull.svg | 70 +-
dev/breeze/doc/images/output_prod-image_pull.txt | 2 +-
dev/breeze/doc/images/output_prod-image_save.svg | 132 ++
dev/breeze/doc/images/output_prod-image_save.txt | 1 +
dev/breeze/doc/images/output_prod-image_verify.svg | 56 +-
dev/breeze/doc/images/output_prod-image_verify.txt | 2 +-
.../doc/images/output_release-management.svg | 20 +-
.../doc/images/output_release-management.txt | 2 +-
...tput_release-management_add-back-references.svg | 74 +-
...tput_release-management_add-back-references.txt | 2 +-
...ase-management_clean-old-provider-artifacts.svg | 8 +-
...tput_release-management_create-minor-branch.svg | 6 +-
...put_release-management_generate-constraints.svg | 58 +-
...put_release-management_generate-constraints.txt | 2 +-
...ease-management_generate-issue-content-core.svg | 20 +-
...anagement_generate-issue-content-helm-chart.svg | 16 +-
...management_generate-issue-content-providers.svg | 78 +-
...management_generate-issue-content-providers.txt | 2 +-
...ease-management_generate-providers-metadata.svg | 6 +-
...elease-management_install-provider-packages.svg | 88 +-
...elease-management_install-provider-packages.txt | 2 +-
..._release-management_prepare-airflow-package.svg | 12 +-
..._release-management_prepare-airflow-tarball.svg | 4 +-
...lease-management_prepare-helm-chart-package.svg | 8 +-
...lease-management_prepare-helm-chart-tarball.svg | 20 +-
...e-management_prepare-provider-documentation.svg | 102 +-
...e-management_prepare-provider-documentation.txt | 2 +-
...elease-management_prepare-provider-packages.svg | 90 +-
...elease-management_prepare-provider-packages.txt | 2 +-
...ut_release-management_prepare-python-client.svg | 18 +-
...release-management_prepare-task-sdk-package.svg | 116 ++
...release-management_prepare-task-sdk-package.txt | 1 +
.../output_release-management_publish-docs.svg | 110 +-
.../output_release-management_publish-docs.txt | 2 +-
...tput_release-management_release-prod-images.svg | 2 +-
...tput_release-management_release-prod-images.txt | 2 +-
.../output_release-management_start-rc-process.svg | 20 +-
.../output_release-management_start-rc-process.txt | 2 +-
.../output_release-management_start-release.svg | 8 +-
.../output_release-management_tag-providers.svg | 8 +-
...utput_release-management_update-constraints.svg | 22 +-
...release-management_verify-provider-packages.svg | 86 +-
...release-management_verify-provider-packages.txt | 2 +-
dev/breeze/doc/images/output_sbom.svg | 18 +-
.../output_sbom_build-all-airflow-images.svg | 20 +-
.../output_sbom_export-dependency-information.svg | 86 +-
.../output_sbom_export-dependency-information.txt | 2 +-
...output_sbom_generate-providers-requirements.svg | 96 +-
...output_sbom_generate-providers-requirements.txt | 2 +-
.../output_setup_check-all-params-in-groups.svg | 76 +-
.../output_setup_check-all-params-in-groups.txt | 2 +-
dev/breeze/doc/images/output_setup_config.svg | 26 +-
dev/breeze/doc/images/output_setup_config.txt | 2 +-
.../output_setup_regenerate-command-images.svg | 86 +-
.../output_setup_regenerate-command-images.txt | 2 +-
.../doc/images/output_setup_self-upgrade.svg | 4 +-
.../output_setup_synchronize-local-mounts.svg | 6 +-
dev/breeze/doc/images/output_setup_version.svg | 6 +-
dev/breeze/doc/images/output_shell.svg | 254 ++--
dev/breeze/doc/images/output_shell.txt | 2 +-
dev/breeze/doc/images/output_start-airflow.svg | 210 ++-
dev/breeze/doc/images/output_start-airflow.txt | 2 +-
dev/breeze/doc/images/output_static-checks.svg | 182 ++-
dev/breeze/doc/images/output_static-checks.txt | 2 +-
dev/breeze/doc/images/output_testing.svg | 44 +-
dev/breeze/doc/images/output_testing.txt | 2 +-
.../output_testing_core-integration-tests.svg | 256 ++++
.../output_testing_core-integration-tests.txt | 1 +
.../doc/images/output_testing_core-tests.svg | 484 +++++++
.../doc/images/output_testing_core-tests.txt | 1 +
dev/breeze/doc/images/output_testing_db-tests.svg | 504 -------
dev/breeze/doc/images/output_testing_db-tests.txt | 1 -
.../images/output_testing_docker-compose-tests.svg | 44 +-
.../images/output_testing_docker-compose-tests.txt | 2 +-
.../doc/images/output_testing_helm-tests.svg | 44 +-
.../doc/images/output_testing_helm-tests.txt | 2 +-
.../images/output_testing_integration-tests.svg | 252 ----
.../images/output_testing_integration-tests.txt | 1 -
.../doc/images/output_testing_non-db-tests.svg | 464 -------
.../doc/images/output_testing_non-db-tests.txt | 1 -
.../output_testing_providers-integration-tests.svg | 260 ++++
.../output_testing_providers-integration-tests.txt | 1 +
.../doc/images/output_testing_providers-tests.svg | 524 +++++++
.../doc/images/output_testing_providers-tests.txt | 1 +
.../output_testing_python-api-client-tests.svg | 224 +++
.../output_testing_python-api-client-tests.txt | 1 +
.../doc/images/output_testing_system-tests.svg | 240 ++++
.../doc/images/output_testing_system-tests.txt | 1 +
.../doc/images/output_testing_task-sdk-tests.svg | 196 +++
.../doc/images/output_testing_task-sdk-tests.txt | 1 +
dev/breeze/doc/images/output_testing_tests.svg | 576 --------
dev/breeze/doc/images/output_testing_tests.txt | 1 -
dev/breeze/pyproject.toml | 6 +-
.../src/airflow_breeze/commands/ci_commands.py | 15 +-
.../airflow_breeze/commands/ci_image_commands.py | 312 ++++-
.../commands/ci_image_commands_config.py | 54 +-
.../commands/common_image_options.py | 65 +-
.../src/airflow_breeze/commands/common_options.py | 71 +-
.../airflow_breeze/commands/developer_commands.py | 135 +-
.../commands/developer_commands_config.py | 23 +-
.../airflow_breeze/commands/kubernetes_commands.py | 104 +-
.../commands/kubernetes_commands_config.py | 2 -
.../src/airflow_breeze/commands/main_command.py | 6 +-
.../commands/minor_release_command.py | 2 +-
.../commands/production_image_commands.py | 152 ++-
.../commands/production_image_commands_config.py | 33 +-
.../commands/release_candidate_command.py | 14 +-
.../commands/release_management_commands.py | 279 +++-
.../commands/release_management_commands_config.py | 20 +-
.../src/airflow_breeze/commands/sbom_commands.py | 318 ++++-
.../commands/sbom_commands_config.py | 26 +-
.../src/airflow_breeze/commands/setup_commands.py | 19 +-
.../commands/setup_commands_config.py | 1 -
.../airflow_breeze/commands/testing_commands.py | 857 ++++++++----
.../commands/testing_commands_config.py | 423 +++---
.../src/airflow_breeze/configure_rich_click.py | 2 +-
dev/breeze/src/airflow_breeze/global_constants.py | 212 ++-
.../src/airflow_breeze/params/build_ci_params.py | 2 -
.../src/airflow_breeze/params/build_prod_params.py | 18 +-
.../airflow_breeze/params/common_build_params.py | 32 +-
.../src/airflow_breeze/params/shell_params.py | 103 +-
dev/breeze/src/airflow_breeze/pre_commit_ids.py | 16 +-
.../prepare_providers/provider_documentation.py | 193 ++-
.../prepare_providers/provider_packages.py | 32 +-
.../provider_issue_TEMPLATE.md.jinja2 | 4 +-
.../templates/CHANGELOG_TEMPLATE.rst.jinja2 | 10 +
.../PROVIDER_CHANGELOG_TEMPLATE.rst.jinja2 | 3 +-
.../templates/PROVIDER_COMMITS_TEMPLATE.rst.jinja2 | 5 +-
.../templates/PROVIDER_README_TEMPLATE.rst.jinja2 | 3 +-
.../templates/get_provider_info_TEMPLATE.py.jinja2 | 3 +-
.../templates/pyproject_TEMPLATE.toml.jinja2 | 7 +-
.../src/airflow_breeze/utils/backtracking.py | 2 +-
dev/breeze/src/airflow_breeze/utils/black_utils.py | 4 +-
dev/breeze/src/airflow_breeze/utils/cdxgen.py | 124 +-
dev/breeze/src/airflow_breeze/utils/coertions.py | 2 +-
dev/breeze/src/airflow_breeze/utils/console.py | 9 +-
.../src/airflow_breeze/utils/custom_param_types.py | 3 +-
.../airflow_breeze/utils/docker_command_utils.py | 24 +-
.../airflow_breeze/utils/functools_cache.py} | 15 +-
dev/breeze/src/airflow_breeze/utils/github.py | 126 ++
dev/breeze/src/airflow_breeze/utils/image.py | 59 +-
.../src/airflow_breeze/utils/kubernetes_utils.py | 7 +-
.../utils/mark_image_as_refreshed.py | 2 +-
dev/breeze/src/airflow_breeze/utils/packages.py | 219 ++-
dev/breeze/src/airflow_breeze/utils/parallel.py | 3 +-
dev/breeze/src/airflow_breeze/utils/path_utils.py | 29 +-
dev/breeze/src/airflow_breeze/utils/platforms.py | 6 +-
.../utils/projects_google_spreadsheet.py | 252 ++++
.../airflow_breeze/utils/provider_dependencies.py | 9 +-
.../airflow_breeze/utils/publish_docs_helpers.py | 36 +-
.../src/airflow_breeze/utils/python_versions.py | 13 -
dev/breeze/src/airflow_breeze/utils/run_tests.py | 353 ++---
dev/breeze/src/airflow_breeze/utils/run_utils.py | 174 ++-
.../src/airflow_breeze/utils/selective_checks.py | 554 +++++---
.../src/airflow_breeze/utils/spelling_checks.py | 4 +-
.../src/airflow_breeze/utils/version_utils.py | 54 +
dev/breeze/src/airflow_breeze/utils/versions.py | 2 +-
.../src/airflow_breeze/utils/virtualenv_utils.py | 14 +-
dev/breeze/src/airflow_breeze/utils/visuals.py | 3 +
dev/breeze/tests/conftest.py | 9 +
dev/breeze/tests/test_cache.py | 6 +-
dev/breeze/tests/test_docker_command_utils.py | 22 +
dev/breeze/tests/test_packages.py | 133 +-
dev/breeze/tests/test_provider_documentation.py | 32 +-
.../tests/test_pytest_args_for_test_types.py | 331 +++--
dev/breeze/tests/test_run_test_args.py | 151 +++
dev/breeze/tests/test_selective_checks.py | 1423 +++++++++++++-------
dev/breeze/tests/test_shell_params.py | 28 +-
dev/breeze/uv.lock | 639 +++++----
dev/stats/explore_pr_candidates.ipynb | 38 +-
docs/docker-stack/build-arg-ref.rst | 1 -
generated/provider_dependencies.json | 1386 +------------------
pyproject.toml | 4 +-
scripts/ci/cleanup_docker.sh | 5 +-
scripts/ci/constraints/ci_commit_constraints.sh | 3 -
scripts/ci/docker-compose/base-ports.yml | 1 +
scripts/ci/docker-compose/base.yml | 2 +-
scripts/ci/docker-compose/devcontainer.env | 4 -
scripts/ci/docker-compose/forward-credentials.yml | 1 +
...on-openlineage.yml => integration-keycloak.yml} | 59 +-
.../ci/docker-compose/integration-openlineage.yml | 4 +-
.../ci/docker-compose/keycloak/init-keycloak-db.sh | 20 +-
.../docker-compose/keycloak/keycloak-entrypoint.sh | 45 +
scripts/ci/docker-compose/local.yml | 9 +
.../docker-compose/providers-and-tests-sources.yml | 5 +-
.../ci_start_arm_instance_and_connect_to_docker.sh | 91 --
scripts/ci/images/ci_stop_arm_instance.sh | 30 -
scripts/ci/install_breeze.sh | 10 +-
.../pre_commit/base_operator_partial_arguments.py | 80 +-
scripts/ci/pre_commit/boring_cyborg.py | 19 +-
.../check_cncf_k8s_used_for_k8s_executor_only.py | 7 +-
.../ci/pre_commit/check_common_sql_dependency.py | 30 +-
scripts/ci/pre_commit/check_deferrable_default.py | 2 +-
scripts/ci/pre_commit/check_deprecations.py | 59 +-
.../ci/pre_commit/check_imports_in_providers.py | 105 ++
scripts/ci/pre_commit/check_min_python_version.py | 2 +-
scripts/ci/pre_commit/check_pre_commit_hooks.py | 2 +-
scripts/ci/pre_commit/check_provider_yaml_files.py | 15 +-
.../check_providers_subpackages_all_have_init.py | 44 +-
scripts/ci/pre_commit/check_system_tests.py | 10 +-
.../check_system_tests_hidden_in_index.py | 4 +-
...ation_reference.py => check_template_fields.py} | 19 +-
.../ci/pre_commit/check_tests_in_right_folders.py | 3 +-
.../ci/pre_commit/check_ti_vs_tis_attributes.py | 5 +-
scripts/ci/pre_commit/checkout_no_credentials.py | 7 +
.../ci/pre_commit/common_precommit_black_utils.py | 4 +-
scripts/ci/pre_commit/common_precommit_utils.py | 45 +
scripts/ci/pre_commit/compat_cache_on_methods.py | 69 -
...{compile_www_assets.py => compile_ui_assets.py} | 39 +-
...pile_www_assets.py => compile_ui_assets_dev.py} | 65 +-
scripts/ci/pre_commit/compile_www_assets.py | 17 +-
.../decorator_operator_implements_custom_name.py | 2 +-
scripts/ci/pre_commit/generate_airflow_diagrams.py | 14 +-
scripts/ci/pre_commit/helm_lint.py | 2 +-
scripts/ci/pre_commit/kubeconform.py | 2 +-
scripts/ci/pre_commit/{www_lint.py => lint_ui.py} | 13 +-
scripts/ci/pre_commit/{www_lint.py => lint_www.py} | 0
scripts/ci/pre_commit/migration_reference.py | 14 +-
scripts/ci/pre_commit/mypy_folder.py | 29 +-
scripts/ci/pre_commit/sync_init_decorator.py | 206 +--
scripts/ci/pre_commit/update_build_dependencies.py | 110 --
.../ci/pre_commit/update_common_sql_api_stubs.py | 8 +-
scripts/ci/pre_commit/update_er_diagram.py | 13 +-
scripts/ci/pre_commit/update_example_dags_paths.py | 13 +-
...roviders_init.py => update_fastapi_api_spec.py} | 23 +-
scripts/ci/pre_commit/update_installers.py | 152 ---
.../pre_commit/update_installers_and_pre_commit.py | 218 +++
.../ci/pre_commit/update_providers_build_files.py | 112 ++
.../ci/pre_commit/update_providers_dependencies.py | 118 +-
scripts/ci/pre_commit/validate_operators_init.py | 8 +-
scripts/ci/pre_commit/vendor_k8s_json_schema.py | 2 +-
scripts/ci/pre_commit/version_heads_map.py | 82 +-
...retry.sh => run_breeze_command_with_retries.sh} | 40 +-
.../ci/testing/run_integration_tests_with_retry.sh | 15 +-
.../run_system_tests.sh} | 18 +-
scripts/ci/testing/run_unit_tests.sh | 151 +++
scripts/ci/testing/summarize_captured_warnings.py | 6 +-
scripts/ci/testing/summarize_junit_failures.py | 4 +-
.../providers/papermill/input_notebook.ipynb | 2 +-
367 files changed, 15491 insertions(+), 12197 deletions(-)
diff --git a/.dockerignore b/.dockerignore
index dba7378a3b7..197f14a0369 100644
--- a/.dockerignore
+++ b/.dockerignore
@@ -34,12 +34,15 @@
!chart
!docs
!licenses
+!providers/
+!task_sdk/
# Add those folders to the context so that they are available in the CI container
!scripts
# Add tests and kubernetes_tests to context.
!tests
+!tests_common
!kubernetes_tests
!helm_tests
!docker_tests
@@ -79,6 +82,7 @@
airflow/git_version
# Exclude mode_modules pulled by "yarn" for compilation of www files generated by NPM
airflow/www/node_modules
+airflow/ui/node_modules
# Exclude link to docs
airflow/www/static/docs
@@ -121,6 +125,9 @@ docs/_build/
docs/_api/
docs/_doctrees/
+# Exclude new providers docs generated files
+providers/**/docs/_api/
+
# files generated by memray
*.py.*.html
*.py.*.bin
diff --git a/.github/actions/breeze/action.yml b/.github/actions/breeze/action.yml
index 164914c3d52..d7eaa1b088b 100644
--- a/.github/actions/breeze/action.yml
+++ b/.github/actions/breeze/action.yml
@@ -21,10 +21,10 @@ description: 'Sets up Python and Breeze'
inputs:
python-version:
description: 'Python version to use'
- # Version of Python used for reproducibility of the packages built
- # Python 3.8 tarfile produces different tarballs than Python 3.9+ tarfile that's why we are forcing
- # Python 3.9 for all release preparation commands to make sure that the tarballs are reproducible
default: "3.9"
+ use-uv:
+ description: 'Whether to use uv tool'
+ required: true
outputs:
host-python-version:
description: Python version used in host
@@ -36,13 +36,11 @@ runs:
uses: actions/setup-python@v5
with:
python-version: ${{ inputs.python-version }}
- cache: 'pip'
- cache-dependency-path: ./dev/breeze/pyproject.toml
+ # NOTE! Installing Breeze without using cache is FASTER than when using cache - uv is so fast and has
+ # so low overhead, that just running upload cache/restore cache is slower than installing it from scratch
- name: "Install Breeze"
shell: bash
run: ./scripts/ci/install_breeze.sh
- env:
- PYTHON_VERSION: ${{ inputs.python-version }}
- name: "Free space"
shell: bash
run: breeze ci free-space
diff --git a/.github/actions/checkout_target_commit/action.yml b/.github/actions/checkout_target_commit/action.yml
deleted file mode 100644
index e95e8b86254..00000000000
--- a/.github/actions/checkout_target_commit/action.yml
+++ /dev/null
@@ -1,81 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
----
-name: 'Checkout target commit'
-description: >
- Checks out target commit with the exception of .github scripts directories that come from the target branch
-inputs:
- target-commit-sha:
- description: 'SHA of the target commit to checkout'
- required: true
- pull-request-target:
- description: 'Whether the workflow is a pull request target workflow'
- required: true
- is-committer-build:
- description: 'Whether the build is done by a committer'
- required: true
-runs:
- using: "composite"
- steps:
- - name: "Checkout target commit"
- uses: actions/checkout@v4
- with:
- ref: ${{ inputs.target-commit-sha }}
- persist-credentials: false
- ####################################################################################################
- # BE VERY CAREFUL HERE! THIS LINE AND THE END OF THE WARNING. IN PULL REQUEST TARGET WORKFLOW
- # WE CHECK OUT THE TARGET COMMIT ABOVE TO BE ABLE TO BUILD THE IMAGE FROM SOURCES FROM THE
- # INCOMING PR, RATHER THAN FROM TARGET BRANCH. THIS IS A SECURITY RISK, BECAUSE THE PR
- # CAN CONTAIN ANY CODE AND WE EXECUTE IT HERE. THEREFORE, WE NEED TO BE VERY CAREFUL WHAT WE
- # DO HERE. WE SHOULD NOT EXECUTE ANY CODE THAT COMES FROM THE PR. WE SHOULD NOT RUN ANY BREEZE
- # COMMAND NOR SCRIPTS NOR COMPOSITE ACTIONS. WE SHOULD ONLY RUN CODE THAT IS EMBEDDED DIRECTLY IN
- # THIS WORKFLOW - BECAUSE THIS IS THE ONLY CODE THAT WE CAN TRUST.
- ####################################################################################################
- - name: Checkout target branch to 'target-airflow' folder to use ci/scripts and breeze from there.
- uses: actions/checkout@v4
- with:
- path: "target-airflow"
- ref: ${{ github.base_ref }}
- persist-credentials: false
- if: inputs.pull-request-target == 'true' && inputs.is-committer-build != 'true'
- - name: >
- Replace "scripts/ci", "dev", ".github/actions" and ".github/workflows" with the target branch
- so that the those directories are not coming from the PR
- shell: bash
- run: |
- echo
- echo -e "\033[33m Replace scripts, dev, actions with target branch for non-committer builds!\033[0m"
- echo
- rm -rfv "scripts/ci"
- rm -rfv "dev"
- rm -rfv ".github/actions"
- rm -rfv ".github/workflows"
- rm -v ".dockerignore" || true
- mv -v "target-airflow/scripts/ci" "scripts"
- mv -v "target-airflow/dev" "."
- mv -v "target-airflow/.github/actions" "target-airflow/.github/workflows" ".github"
- mv -v "target-airflow/.dockerignore" ".dockerignore" || true
- if: inputs.pull-request-target == 'true' && inputs.is-committer-build != 'true'
- ####################################################################################################
- # AFTER IT'S SAFE. THE `dev`, `scripts/ci` AND `.github/actions` and `.dockerignore` ARE NOW COMING
- # FROM THE BASE_REF - WHICH IS THE TARGET BRANCH OF THE PR. WE CAN TRUST THAT THOSE SCRIPTS ARE
- # SAFE TO RUN AND CODE AVAILABLE IN THE DOCKER BUILD PHASE IS CONTROLLED BY THE `.dockerignore`.
- # ALL THE REST OF THE CODE COMES FROM THE PR, AND FOR EXAMPLE THE CODE IN THE `Dockerfile.ci` CAN
- # BE RUN SAFELY AS PART OF DOCKER BUILD. BECAUSE IT RUNS INSIDE THE DOCKER CONTAINER AND IT IS
- # ISOLATED FROM THE RUNNER.
- ####################################################################################################
diff --git a/.github/actions/install-pre-commit/action.yml b/.github/actions/install-pre-commit/action.yml
index 57624f48e89..92147d9ee02 100644
--- a/.github/actions/install-pre-commit/action.yml
+++ b/.github/actions/install-pre-commit/action.yml
@@ -21,30 +21,58 @@ description: 'Installs pre-commit and related packages'
inputs:
python-version:
description: 'Python version to use'
- default: 3.9
+ default: "3.9"
uv-version:
description: 'uv version to use'
- default: 0.5.11
+ default: "0.5.14" # Keep this comment to allow automatic replacement of uv version
pre-commit-version:
description: 'pre-commit version to use'
- default: 4.0.1
+ default: "4.0.1" # Keep this comment to allow automatic replacement of pre-commit version
pre-commit-uv-version:
description: 'pre-commit-uv version to use'
- default: 4.1.4
+ default: "4.1.4" # Keep this comment to allow automatic replacement of pre-commit-uv version
runs:
using: "composite"
steps:
- name: Install pre-commit, uv, and pre-commit-uv
shell: bash
- run: >
- pip install
- pre-commit==${{inputs.pre-commit-version}}
- uv==${{inputs.uv-version}}
- pre-commit-uv==${{inputs.pre-commit-uv-version}}
- - name: Cache pre-commit envs
- uses: actions/cache@v4
+ env:
+ UV_VERSION: ${{inputs.uv-version}}
+ PRE_COMMIT_VERSION: ${{inputs.pre-commit-version}}
+ PRE_COMMIT_UV_VERSION: ${{inputs.pre-commit-uv-version}}
+ run: |
+ pip install uv==${UV_VERSION} || true
+ uv tool install pre-commit==${PRE_COMMIT_VERSION} --with uv==${UV_VERSION} \
+ --with pre-commit-uv==${PRE_COMMIT_UV_VERSION}
+ working-directory: ${{ github.workspace }}
+ # We need to use tar file with archive to restore all the permissions and symlinks
+ - name: "Delete ~.cache"
+ run: |
+ du ~/ --max-depth=2
+ echo
+ echo Deleting ~/.cache
+ echo
+ rm -rf ~/.cache
+ echo
+ shell: bash
+ - name: "Restore pre-commit cache"
+ uses: apache/infrastructure-actions/stash/restore@c94b890bbedc2fc61466d28e6bd9966bc6c6643c
with:
- path: ~/.cache/pre-commit
- key: "pre-commit-${{inputs.python-version}}-${{ hashFiles('.pre-commit-config.yaml') }}"
- restore-keys: |
- pre-commit-${{inputs.python-version}}-
+ key: cache-pre-commit-v4-${{ inputs.python-version }}-${{ hashFiles('.pre-commit-config.yaml') }}
+ path: /tmp/
+ id: restore-pre-commit-cache
+ - name: "Restore .cache from the tar file"
+ run: tar -C ~ -xzf /tmp/cache-pre-commit.tar.gz
+ shell: bash
+ if: steps.restore-pre-commit-cache.outputs.stash-hit == 'true'
+ - name: "Show restored files"
+ run: |
+ echo "Restored files"
+ du ~/ --max-depth=2
+ echo
+ shell: bash
+ if: steps.restore-pre-commit-cache.outputs.stash-hit == 'true'
+ - name: Install pre-commit hooks
+ shell: bash
+ run: pre-commit install-hooks || (cat ~/.cache/pre-commit/pre-commit.log && exit 1)
+ working-directory: ${{ github.workspace }}
diff --git a/.github/actions/migration_tests/action.yml b/.github/actions/migration_tests/action.yml
index a1d098713a2..7f27ca81cd0 100644
--- a/.github/actions/migration_tests/action.yml
+++ b/.github/actions/migration_tests/action.yml
@@ -45,6 +45,20 @@ runs:
COMPOSE_PROJECT_NAME: "docker-compose"
env:
COMPOSE_PROJECT_NAME: "docker-compose"
+ - name: "Bring compose down again"
+ shell: bash
+ run: breeze down
+ env:
+ COMPOSE_PROJECT_NAME: "docker-compose"
+ - name: "Test offline migration ${{env.BACKEND}}"
+ shell: bash
+ run: >
+ breeze shell "airflow db reset -y &&
+ airflow db downgrade -n 2.7.0 -y &&
+ airflow db migrate -s"
+ env:
+ COMPOSE_PROJECT_NAME: "docker-compose"
+ if: env.BACKEND != 'sqlite'
- name: "Bring any containers left down"
shell: bash
run: breeze down
diff --git a/.github/actions/post_tests_success/action.yml b/.github/actions/post_tests_success/action.yml
index 37b51154d3e..b7b00a6fc0d 100644
--- a/.github/actions/post_tests_success/action.yml
+++ b/.github/actions/post_tests_success/action.yml
@@ -33,7 +33,7 @@ runs:
- name: "Upload artifact for warnings"
uses: actions/upload-artifact@v4
with:
- name: test-warnings-${{env.JOB_ID}}
+ name: test-warnings-${{ env.JOB_ID }}
path: ./files/warnings-*.txt
retention-days: 7
if-no-files-found: ignore
@@ -50,5 +50,5 @@ runs:
if: env.ENABLE_COVERAGE == 'true' && env.TEST_TYPES != 'Helm' && inputs.python-version != '3.12'
with:
name: coverage-${{env.JOB_ID}}
- flags: python-${{env.PYTHON_MAJOR_MINOR_VERSION}},${{env.BACKEND}}-${{env.BACKEND_VERSION}}
+ flags: python-${{ env.PYTHON_MAJOR_MINOR_VERSION }},${{ env.BACKEND }}-${{ env.BACKEND_VERSION }}
directory: "./files/coverage-reports/"
diff --git a/.github/actions/prepare_all_ci_images/action.yml b/.github/actions/prepare_all_ci_images/action.yml
new file mode 100644
index 00000000000..d156818b9b2
--- /dev/null
+++ b/.github/actions/prepare_all_ci_images/action.yml
@@ -0,0 +1,68 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: 'Prepare all CI images'
+description: 'Recreates current python CI images from artifacts for all python versions'
+inputs:
+ python-versions-list-as-string:
+ description: 'Stringified array of all Python versions to test - separated by spaces.'
+ required: true
+ platform:
+ description: 'Platform for the build - linux/amd64 or linux/arm64'
+ required: true
+runs:
+ using: "composite"
+ steps:
+ - name: "Cleanup docker"
+ run: ./scripts/ci/cleanup_docker.sh
+ shell: bash
+ # TODO: Currently we cannot loop through the list of python versions and have dynamic list of
+ # tasks. Instead we hardcode all possible python versions and they - but
+ # this should be implemented in stash action as list of keys to download.
+ # That includes 3.8 - 3.12 as we are backporting it to v2-10-test branch
+ # This is captured in https://github.com/apache/airflow/issues/45268
+ - name: "Restore CI docker image ${{ inputs.platform }}:3.8"
+ uses: ./.github/actions/prepare_single_ci_image
+ with:
+ platform: ${{ inputs.platform }}
+ python: "3.8"
+ python-versions-list-as-string: ${{ inputs.python-versions-list-as-string }}
+ - name: "Restore CI docker image ${{ inputs.platform }}:3.9"
+ uses: ./.github/actions/prepare_single_ci_image
+ with:
+ platform: ${{ inputs.platform }}
+ python: "3.9"
+ python-versions-list-as-string: ${{ inputs.python-versions-list-as-string }}
+ - name: "Restore CI docker image ${{ inputs.platform }}:3.10"
+ uses: ./.github/actions/prepare_single_ci_image
+ with:
+ platform: ${{ inputs.platform }}
+ python: "3.10"
+ python-versions-list-as-string: ${{ inputs.python-versions-list-as-string }}
+ - name: "Restore CI docker image ${{ inputs.platform }}:3.11"
+ uses: ./.github/actions/prepare_single_ci_image
+ with:
+ platform: ${{ inputs.platform }}
+ python: "3.11"
+ python-versions-list-as-string: ${{ inputs.python-versions-list-as-string }}
+ - name: "Restore CI docker image ${{ inputs.platform }}:3.12"
+ uses: ./.github/actions/prepare_single_ci_image
+ with:
+ platform: ${{ inputs.platform }}
+ python: "3.12"
+ python-versions-list-as-string: ${{ inputs.python-versions-list-as-string }}
diff --git a/.github/actions/prepare_breeze_and_image/action.yml b/.github/actions/prepare_breeze_and_image/action.yml
index 41aa17092d5..e6755444b2f 100644
--- a/.github/actions/prepare_breeze_and_image/action.yml
+++ b/.github/actions/prepare_breeze_and_image/action.yml
@@ -16,12 +16,21 @@
# under the License.
#
---
-name: 'Prepare breeze && current python image'
-description: 'Installs breeze and pulls current python image'
+name: 'Prepare breeze && current image (CI or PROD)'
+description: 'Installs breeze and recreates current python image from artifact'
inputs:
- pull-image-type:
- description: 'Which image to pull'
- default: CI
+ python:
+ description: 'Python version for image to prepare'
+ required: true
+ image-type:
+ description: 'Which image type to prepare (ci/prod)'
+ default: "ci"
+ platform:
+ description: 'Platform for the build - linux/amd64 or linux/arm64'
+ required: true
+ use-uv:
+ description: 'Whether to use uv'
+ required: true
outputs:
host-python-version:
description: Python version used in host
@@ -29,17 +38,25 @@ outputs:
runs:
using: "composite"
steps:
+ - name: "Cleanup docker"
+ run: ./scripts/ci/cleanup_docker.sh
+ shell: bash
- name: "Install Breeze"
uses: ./.github/actions/breeze
+ with:
+ use-uv: ${{ inputs.use-uv }}
id: breeze
- - name: Login to ghcr.io
- shell: bash
- run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
- - name: Pull CI image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG }}
- shell: bash
- run: breeze ci-image pull --tag-as-latest
- if: inputs.pull-image-type == 'CI'
- - name: Pull PROD image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG }}
+ - name: "Restore ${{ inputs.image-type }} docker image ${{ inputs.platform }}:${{ inputs.python }}"
+ uses: apache/infrastructure-actions/stash/restore@c94b890bbedc2fc61466d28e6bd9966bc6c6643c
+ with:
+ key: ${{ inputs.image-type }}-image-save-${{ inputs.platform }}-${{ inputs.python }}
+ path: "/tmp/"
+ - name: "Load ${{ inputs.image-type }} image ${{ inputs.platform }}:${{ inputs.python }}"
+ env:
+ PLATFORM: ${{ inputs.platform }}
+ PYTHON: ${{ inputs.python }}
+ IMAGE_TYPE: ${{ inputs.image-type }}
+ run: >
+ breeze ${IMAGE_TYPE}-image load
+ --platform ${PLATFORM} --python ${PYTHON}
shell: bash
- run: breeze prod-image pull --tag-as-latest
- if: inputs.pull-image-type == 'PROD'
diff --git a/.github/actions/prepare_single_ci_image/action.yml b/.github/actions/prepare_single_ci_image/action.yml
new file mode 100644
index 00000000000..3dde30033aa
--- /dev/null
+++ b/.github/actions/prepare_single_ci_image/action.yml
@@ -0,0 +1,50 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: 'Prepare single CI image'
+description: >
+ Recreates current python image from artifacts (needed for the hard-coded actions calling all
+ possible Python versions in "prepare_all_ci_images" action. Hopefully we can get rid of it when
+ the https://github.com/apache/airflow/issues/45268 is resolved and we contribute capability of
+ downloading multiple keys to the stash action.
+inputs:
+ python:
+ description: 'Python version for image to prepare'
+ required: true
+ python-versions-list-as-string:
+ description: 'Stringified array of all Python versions to prepare - separated by spaces.'
+ required: true
+ platform:
+ description: 'Platform for the build - linux/amd64 or linux/arm64'
+ required: true
+runs:
+ using: "composite"
+ steps:
+ - name: "Restore CI docker images ${{ inputs.platform }}:${{ inputs.python }}"
+ uses: apache/infrastructure-actions/stash/restore@c94b890bbedc2fc61466d28e6bd9966bc6c6643c
+ with:
+ key: ci-image-save-${{ inputs.platform }}-${{ inputs.python }}
+ path: "/tmp/"
+ if: contains(inputs.python-versions-list-as-string, inputs.python)
+ - name: "Load CI image ${{ inputs.platform }}:${{ inputs.python }}"
+ env:
+ PLATFORM: ${{ inputs.platform }}
+ PYTHON: ${{ inputs.python }}
+ run: breeze ci-image load --platform "${PLATFORM}" --python "${PYTHON}"
+ shell: bash
+ if: contains(inputs.python-versions-list-as-string, inputs.python)
diff --git a/.github/workflows/additional-ci-image-checks.yml b/.github/workflows/additional-ci-image-checks.yml
index 82b143d2f03..56cee169762 100644
--- a/.github/workflows/additional-ci-image-checks.yml
+++ b/.github/workflows/additional-ci-image-checks.yml
@@ -32,10 +32,6 @@ on: # yamllint disable-line rule:truthy
description: "The array of labels (in json form) determining self-hosted runners."
required: true
type: string
- image-tag:
- description: "Tag to set for the image"
- required: true
- type: string
python-versions:
description: "The list of python versions (stringified JSON array) to run the tests on."
required: true
@@ -64,6 +60,10 @@ on: # yamllint disable-line rule:truthy
description: "Docker cache specification to build the image (registry, local, disabled)."
required: true
type: string
+ disable-airflow-repo-cache:
+ description: "Disable airflow repo cache read from main."
+ required: true
+ type: string
canary-run:
description: "Whether this is a canary run (true/false)"
required: true
@@ -99,8 +99,6 @@ jobs:
contents: read
# This write is only given here for `push` events from "apache/airflow" repo. It is not given for PRs
# from forks. This is to prevent malicious PRs from creating images in the "apache/airflow" repo.
- # For regular build for PRS this "build-prod-images" workflow will be skipped anyway by the
- # "in-workflow-build" condition
packages: write
secrets: inherit
with:
@@ -113,9 +111,10 @@ jobs:
python-versions: ${{ inputs.python-versions }}
branch: ${{ inputs.branch }}
constraints-branch: ${{ inputs.constraints-branch }}
- use-uv: ${{ inputs.use-uv}}
+ use-uv: ${{ inputs.use-uv }}
include-success-outputs: ${{ inputs.include-success-outputs }}
docker-cache: ${{ inputs.docker-cache }}
+ disable-airflow-repo-cache: ${{ inputs.disable-airflow-repo-cache }}
if: inputs.branch == 'main'
# Check that after earlier cache push, breeze command will build quickly
@@ -144,8 +143,13 @@ jobs:
run: ./scripts/ci/cleanup_docker.sh
- name: "Install Breeze"
uses: ./.github/actions/breeze
+ with:
+ use-uv: ${{ inputs.use-uv }}
- name: "Login to ghcr.io"
- run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
+ env:
+ actor: ${{ github.actor }}
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ run: echo "$GITHUB_TOKEN" | docker login ghcr.io -u "$actor" --password-stdin
- name: "Check that image builds quickly"
run: breeze shell --max-time 600 --platform "linux/amd64"
@@ -154,21 +158,24 @@ jobs:
# # There is no point in running this one in "canary" run, because the above step is doing the
# # same build anyway.
# build-ci-arm-images:
-# name: Build CI ARM images (in-workflow)
+# name: Build CI ARM images
# uses: ./.github/workflows/ci-image-build.yml
# permissions:
# contents: read
# packages: write
# secrets: inherit
# with:
+# platform: "linux/arm64"
# push-image: "false"
+# upload-image-artifact: "true"
+# upload-mount-cache-artifact: ${{ inputs.canary-run }}
# runs-on-as-json-public: ${{ inputs.runs-on-as-json-public }}
# runs-on-as-json-self-hosted: ${{ inputs.runs-on-as-json-self-hosted }}
-# image-tag: ${{ inputs.image-tag }}
# python-versions: ${{ inputs.python-versions }}
-# platform: "linux/arm64"
# branch: ${{ inputs.branch }}
# constraints-branch: ${{ inputs.constraints-branch }}
-# use-uv: ${{ inputs.use-uv}}
+# use-uv: ${{ inputs.use-uv }}
# upgrade-to-newer-dependencies: ${{ inputs.upgrade-to-newer-dependencies }}
# docker-cache: ${{ inputs.docker-cache }}
+# disable-airflow-repo-cache: ${{ inputs.disable-airflow-repo-cache }}
+#
diff --git a/.github/workflows/additional-prod-image-tests.yml b/.github/workflows/additional-prod-image-tests.yml
index 4c9606e1343..bca5e3a5927 100644
--- a/.github/workflows/additional-prod-image-tests.yml
+++ b/.github/workflows/additional-prod-image-tests.yml
@@ -32,10 +32,6 @@ on: # yamllint disable-line rule:truthy
description: "Branch used to construct constraints URL from."
required: true
type: string
- image-tag:
- description: "Tag to set for the image"
- required: true
- type: string
upgrade-to-newer-dependencies:
description: "Whether to upgrade to newer dependencies (true/false)"
required: true
@@ -48,6 +44,10 @@ on: # yamllint disable-line rule:truthy
description: "Docker cache specification to build the image (registry, local, disabled)."
required: true
type: string
+ disable-airflow-repo-cache:
+ description: "Disable airflow repo cache read from main."
+ required: true
+ type: string
canary-run:
description: "Whether to run the canary run (true/false)"
required: true
@@ -56,6 +56,10 @@ on: # yamllint disable-line rule:truthy
description: "Which version of python should be used by default"
required: true
type: string
+ use-uv:
+ description: "Whether to use uv"
+ required: true
+ type: string
jobs:
prod-image-extra-checks-main:
name: PROD image extra checks (main)
@@ -66,12 +70,12 @@ jobs:
default-python-version: ${{ inputs.default-python-version }}
branch: ${{ inputs.default-branch }}
use-uv: "false"
- image-tag: ${{ inputs.image-tag }}
build-provider-packages: ${{ inputs.default-branch == 'main' }}
upgrade-to-newer-dependencies: ${{ inputs.upgrade-to-newer-dependencies }}
chicken-egg-providers: ${{ inputs.chicken-egg-providers }}
constraints-branch: ${{ inputs.constraints-branch }}
docker-cache: ${{ inputs.docker-cache }}
+ disable-airflow-repo-cache: ${{ inputs.disable-airflow-repo-cache }}
if: inputs.default-branch == 'main' && inputs.canary-run == 'true'
prod-image-extra-checks-release-branch:
@@ -83,12 +87,12 @@ jobs:
default-python-version: ${{ inputs.default-python-version }}
branch: ${{ inputs.default-branch }}
use-uv: "false"
- image-tag: ${{ inputs.image-tag }}
build-provider-packages: ${{ inputs.default-branch == 'main' }}
upgrade-to-newer-dependencies: ${{ inputs.upgrade-to-newer-dependencies }}
chicken-egg-providers: ${{ inputs.chicken-egg-providers }}
constraints-branch: ${{ inputs.constraints-branch }}
docker-cache: ${{ inputs.docker-cache }}
+ disable-airflow-repo-cache: ${{ inputs.disable-airflow-repo-cache }}
if: inputs.default-branch != 'main' && inputs.canary-run == 'true'
test-examples-of-prod-image-building:
@@ -111,36 +115,30 @@ jobs:
persist-credentials: false
- name: "Cleanup docker"
run: ./scripts/ci/cleanup_docker.sh
- - name: "Install Breeze"
- uses: ./.github/actions/breeze
- - name: Login to ghcr.io
- shell: bash
- run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
- - name: Pull PROD image ${{ inputs.default-python-version}}:${{ inputs.image-tag }}
- run: breeze prod-image pull --tag-as-latest
- env:
- PYTHON_MAJOR_MINOR_VERSION: "${{ inputs.default-python-version }}"
- IMAGE_TAG: "${{ inputs.image-tag }}"
- - name: "Setup python"
- uses: actions/setup-python@v5
+ - name: "Prepare breeze & PROD image: ${{ inputs.default-python-version }}"
+ uses: ./.github/actions/prepare_breeze_and_image
with:
- python-version: ${{ inputs.default-python-version }}
- cache: 'pip'
- cache-dependency-path: ./dev/requirements.txt
+ platform: "linux/amd64"
+ image-type: "prod"
+ python: ${{ inputs.default-python-version }}
+ use-uv: ${{ inputs.use-uv }}
- name: "Test examples of PROD image building"
+ env:
+ GITHUB_REPOSITORY: ${{ github.repository }}
+ DEFAULT_BRANCH: ${{ inputs.default-branch }}
+ DEFAULT_PYTHON_VERSION: ${{ inputs.default-python-version }}
run: "
cd ./docker_tests && \
python -m pip install -r requirements.txt && \
- TEST_IMAGE=\"ghcr.io/${{ github.repository }}/${{ inputs.default-branch }}\
- /prod/python${{ inputs.default-python-version }}:${{ inputs.image-tag }}\" \
+ TEST_IMAGE=\"ghcr.io/$GITHUB_REPOSITORY/$DEFAULT_BRANCH\
+ /prod/python$DEFAULT_PYTHON_VERSION\" \
python -m pytest test_examples_of_prod_image_building.py -n auto --color=yes"
test-docker-compose-quick-start:
timeout-minutes: 60
- name: "Docker-compose quick start with PROD image verifying"
+ name: "Docker Compose quick start with PROD image verifying"
runs-on: ${{ fromJSON(inputs.runs-on-as-json-public) }}
env:
- IMAGE_TAG: "${{ inputs.image-tag }}"
PYTHON_MAJOR_MINOR_VERSION: "${{ inputs.default-python-version }}"
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@@ -155,14 +153,13 @@ jobs:
with:
fetch-depth: 2
persist-credentials: false
- - name: "Cleanup docker"
- run: ./scripts/ci/cleanup_docker.sh
- - name: "Install Breeze"
- uses: ./.github/actions/breeze
- - name: Login to ghcr.io
- shell: bash
- run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
- - name: "Pull image ${{ inputs.default-python-version}}:${{ inputs.image-tag }}"
- run: breeze prod-image pull --tag-as-latest
+ - name: "Prepare breeze & PROD image: ${{ env.PYTHON_MAJOR_MINOR_VERSION }}"
+ uses: ./.github/actions/prepare_breeze_and_image
+ with:
+ platform: "linux/amd64"
+ image-type: "prod"
+ python: ${{ env.PYTHON_MAJOR_MINOR_VERSION }}
+ use-uv: ${{ inputs.use-uv }}
+ id: breeze
- name: "Test docker-compose quick start"
run: breeze testing docker-compose-tests
diff --git a/.github/workflows/automatic-backport.yml b/.github/workflows/automatic-backport.yml
new file mode 100644
index 00000000000..b5b22b7491a
--- /dev/null
+++ b/.github/workflows/automatic-backport.yml
@@ -0,0 +1,77 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: Automatic Backport
+on: # yamllint disable-line rule:truthy
+ push:
+ branches:
+ - main
+
+jobs:
+ get-pr-info:
+ name: "Get PR information"
+ runs-on: ubuntu-latest
+ outputs:
+ branches: ${{ steps.pr-info.outputs.branches }}
+ commit-sha: ${{ github.sha }}
+ steps:
+ - name: Get commit SHA
+ id: get-sha
+ run: echo "COMMIT_SHA=${GITHUB_SHA}" >> $GITHUB_ENV
+
+ - name: Find PR information
+ id: pr-info
+ uses: actions/github-script@v7
+ env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ with:
+ script: |
+ const { data: pullRequest } = await github.rest.repos.listPullRequestsAssociatedWithCommit({
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ commit_sha: process.env.GITHUB_SHA
+ });
+ if (pullRequest.length > 0) {
+ const pr = pullRequest[0];
+ const backportBranches = pr.labels
+ .filter(label => label.name.startsWith('backport-to-'))
+ .map(label => label.name.replace('backport-to-', ''));
+
+ console.log(`Commit ${process.env.GITHUB_SHA} is associated with PR ${pr.number}`);
+ console.log(`Backport branches: ${backportBranches}`);
+ core.setOutput('branches', JSON.stringify(backportBranches));
+ } else {
+ console.log('No pull request found for this commit.');
+ core.setOutput('branches', '[]');
+ }
+
+ trigger-backport:
+ name: "Trigger Backport"
+ uses: ./.github/workflows/backport-cli.yml
+ needs: get-pr-info
+ if: ${{ needs.get-pr-info.outputs.branches != '[]' }}
+ strategy:
+ matrix:
+ branch: ${{ fromJSON(needs.get-pr-info.outputs.branches) }}
+ fail-fast: false
+ permissions:
+ contents: write
+ pull-requests: write
+ with:
+ target-branch: ${{ matrix.branch }}
+ commit-sha: ${{ needs.get-pr-info.outputs.commit-sha }}
diff --git a/.github/workflows/backport-cli.yml b/.github/workflows/backport-cli.yml
new file mode 100644
index 00000000000..3706cd65bb0
--- /dev/null
+++ b/.github/workflows/backport-cli.yml
@@ -0,0 +1,122 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: Backport Commit
+on: # yamllint disable-line rule:truthy
+ workflow_dispatch:
+ inputs:
+ commit-sha:
+ description: "Commit sha to backport."
+ required: true
+ type: string
+ target-branch:
+ description: "Target branch to backport."
+ required: true
+ type: string
+
+ workflow_call:
+ inputs:
+ commit-sha:
+ description: "Commit sha to backport."
+ required: true
+ type: string
+ target-branch:
+ description: "Target branch to backport."
+ required: true
+ type: string
+
+permissions:
+ contents: write
+ pull-requests: write
+jobs:
+ backport:
+ runs-on: ubuntu-latest
+
+ steps:
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ id: checkout-for-backport
+ uses: actions/checkout@v4
+ with:
+ persist-credentials: true
+ fetch-depth: 0
+
+ - name: Install Python dependencies
+ run: |
+ python -m pip install --upgrade pip
+ python -m pip install cherry-picker==2.4.0 requests==2.32.3
+
+ - name: Run backport script
+ id: execute-backport
+ env:
+ GH_AUTH: ${{ secrets.GITHUB_TOKEN }}
+ TARGET_BRANCH: ${{ inputs.target-branch }}
+ COMMIT_SHA: ${{ inputs.commit-sha }}
+ run: |
+ git config --global user.email "name@example.com"
+ git config --global user.name "Your Name"
+ set +e
+ {
+ echo 'cherry_picker_output<<EOF'
+ cherry_picker ${COMMIT_SHA} ${TARGET_BRANCH}
+ echo EOF
+ } >> "${GITHUB_OUTPUT}"
+ continue-on-error: true
+
+ - name: Parse backport output
+ id: parse-backport-output
+ env:
+ CHERRY_PICKER_OUTPUT: ${{ steps.execute-backport.outputs.cherry_picker_output }}
+ run: |
+ set +e
+ echo "${CHERRY_PICKER_OUTPUT}"
+
+ url=$(echo "${CHERRY_PICKER_OUTPUT}" | \
+ grep -o 'Backport PR created at https://[^ ]*' | \
+ awk '{print $5}')
+
+ url=${url:-"EMPTY"}
+ if [ "$url" == "EMPTY" ]; then
+ # If the backport failed, abort the workflow
+ cherry_picker --abort
+ fi
+ echo "backport-url=$url" >> "${GITHUB_OUTPUT}"
+ continue-on-error: true
+
+ - name: Update Status
+ id: backport-status
+ env:
+ GH_TOKEN: ${{ github.token }}
+ REPOSITORY: ${{ github.repository }}
+ RUN_ID: ${{ github.run_id }}
+ COMMIT_SHA: ${{ inputs.commit-sha }}
+ TARGET_BRANCH: ${{ inputs.target-branch }}
+ BACKPORT_URL: ${{ steps.parse-backport-output.outputs.backport-url }}
+ run: |
+ COMMIT_INFO_URL="https://api.github.com/repos/$REPOSITORY/commits/"
+ COMMIT_INFO_URL="${COMMIT_INFO_URL}$COMMIT_SHA/pulls"
+
+ PR_NUMBER=$(gh api \
+ -H "Accept: application/vnd.github+json" \
+ -H "X-GitHub-Api-Version: 2022-11-28" \
+ /repos/$REPOSITORY/commits/$COMMIT_SHA/pulls \
+ --jq '.[0].number')
+
+ python ./dev/backport/update_backport_status.py \
+ $BACKPORT_URL \
+ $COMMIT_SHA $TARGET_BRANCH \
+ "$PR_NUMBER"
diff --git a/.github/workflows/basic-tests.yml b/.github/workflows/basic-tests.yml
index 5141feae223..353f65d9a6c 100644
--- a/.github/workflows/basic-tests.yml
+++ b/.github/workflows/basic-tests.yml
@@ -24,6 +24,10 @@ on: # yamllint disable-line rule:truthy
description: "The array of labels (in json form) determining public runners."
required: true
type: string
+ run-ui-tests:
+ description: "Whether to run UI tests (true/false)"
+ required: true
+ type: string
run-www-tests:
description: "Whether to run WWW tests (true/false)"
required: true
@@ -52,12 +56,10 @@ on: # yamllint disable-line rule:truthy
description: "Whether to run only latest version checks (true/false)"
required: true
type: string
- enable-aip-44:
- description: "Whether to enable AIP-44 (true/false)"
+ use-uv:
+ description: "Whether to use uv in the image"
required: true
type: string
-env:
- AIRFLOW_ENABLE_AIP_44: "${{ inputs.enable-aip-44 }}"
jobs:
run-breeze-tests:
timeout-minutes: 10
@@ -74,20 +76,17 @@ jobs:
persist-credentials: false
- name: "Cleanup docker"
run: ./scripts/ci/cleanup_docker.sh
- - uses: actions/setup-python@v5
+ - name: "Install Breeze"
+ uses: ./.github/actions/breeze
with:
- python-version: "${{ inputs.default-python-version }}"
- cache: 'pip'
- cache-dependency-path: ./dev/breeze/pyproject.toml
- - run: pip install --editable ./dev/breeze/
- - run: python -m pytest -n auto --color=yes
+ use-uv: ${{ inputs.use-uv }}
+ - run: uv tool run --from apache-airflow-breeze pytest -n auto --color=yes
working-directory: ./dev/breeze/
-
- tests-www:
+ tests-ui:
timeout-minutes: 10
- name: React WWW tests
+ name: React UI tests
runs-on: ${{ fromJSON(inputs.runs-on-as-json-public) }}
- if: inputs.run-www-tests == 'true'
+ if: inputs.run-ui-tests == 'true'
steps:
- name: "Cleanup repo"
shell: bash
@@ -98,25 +97,41 @@ jobs:
persist-credentials: false
- name: "Cleanup docker"
run: ./scripts/ci/cleanup_docker.sh
+ - name: Setup pnpm
+ uses: pnpm/action-setup@v4.0.0
+ with:
+ version: 9
+ run_install: false
- name: "Setup node"
uses: actions/setup-node@v4
with:
node-version: 21
- - name: "Cache eslint"
- uses: actions/cache@v4
+ cache: 'pnpm'
+ cache-dependency-path: 'airflow/ui/pnpm-lock.yaml'
+ - name: "Restore eslint cache (ui)"
+ uses: apache/infrastructure-actions/stash/restore@c94b890bbedc2fc61466d28e6bd9966bc6c6643c
with:
- path: 'airflow/www/node_modules'
- key: ${{ runner.os }}-www-node-modules-${{ hashFiles('airflow/www/**/yarn.lock') }}
- - run: yarn --cwd airflow/www/ install --frozen-lockfile --non-interactive
- - run: yarn --cwd airflow/www/ run test
+ path: airflow/ui/node_modules/
+ key: cache-ui-node-modules-v1-${{ runner.os }}-${{ hashFiles('airflow/ui/**/pnpm-lock.yaml') }}
+ id: restore-eslint-cache
+ - run: cd airflow/ui && pnpm install --frozen-lockfile
+ - run: cd airflow/ui && pnpm test
env:
FORCE_COLOR: 2
+ - name: "Save eslint cache (ui)"
+ uses: apache/infrastructure-actions/stash/save@c94b890bbedc2fc61466d28e6bd9966bc6c6643c
+ with:
+ path: airflow/ui/node_modules/
+ key: cache-ui-node-modules-v1-${{ runner.os }}-${{ hashFiles('airflow/ui/**/pnpm-lock.yaml') }}
+ if-no-files-found: 'error'
+ retention-days: '2'
+ if: steps.restore-eslint-cache.outputs.stash-hit != 'true'
- test-openapi-client:
+ tests-www:
timeout-minutes: 10
- name: "Test OpenAPI client"
+ name: React WWW tests
runs-on: ${{ fromJSON(inputs.runs-on-as-json-public) }}
- if: inputs.needs-api-codegen == 'true'
+ if: inputs.run-www-tests == 'true'
steps:
- name: "Cleanup repo"
shell: bash
@@ -124,80 +139,57 @@ jobs:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v4
with:
- fetch-depth: 2
persist-credentials: false
- name: "Cleanup docker"
run: ./scripts/ci/cleanup_docker.sh
+ - name: "Setup node"
+ uses: actions/setup-node@v4
+ with:
+ node-version: 21
+ - name: "Restore eslint cache (www)"
+ uses: apache/infrastructure-actions/stash/restore@c94b890bbedc2fc61466d28e6bd9966bc6c6643c
+ with:
+ path: airflow/www/node_modules/
+ key: cache-www-node-modules-v1-${{ runner.os }}-${{ hashFiles('airflow/www/**/yarn.lock') }}
+ id: restore-eslint-cache
+ - run: yarn --cwd airflow/www/ install --frozen-lockfile --non-interactive
+ - run: yarn --cwd airflow/www/ run test
+ env:
+ FORCE_COLOR: 2
+ - name: "Save eslint cache (www)"
+ uses: apache/infrastructure-actions/stash/save@c94b890bbedc2fc61466d28e6bd9966bc6c6643c
+ with:
+ path: airflow/www/node_modules/
+ key: cache-www-node-modules-v1-${{ runner.os }}-${{ hashFiles('airflow/www/**/yarn.lock') }}
+ if-no-files-found: 'error'
+ retention-days: '2'
+ if: steps.restore-eslint-cache.outputs.stash-hit != 'true'
+
+ install-pre-commit:
+ timeout-minutes: 5
+ name: "Install pre-commit for cache"
+ runs-on: ${{ fromJSON(inputs.runs-on-as-json-public) }}
+ env:
+ PYTHON_MAJOR_MINOR_VERSION: "${{ inputs.default-python-version }}"
+ if: inputs.basic-checks-only == 'true'
+ steps:
+ - name: "Cleanup repo"
+ shell: bash
+ run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v4
with:
- repository: "apache/airflow-client-python"
- fetch-depth: 1
persist-credentials: false
- path: ./airflow-client-python
- name: "Install Breeze"
uses: ./.github/actions/breeze
- - name: "Generate client with breeze"
- run: >
- breeze release-management prepare-python-client --package-format both
- --version-suffix-for-pypi dev0 --python-client-repo ./airflow-client-python
- - name: "Show diff"
- run: git diff --color HEAD
- working-directory: ./airflow-client-python
- - name: Install hatch
- run: |
- python -m pip install --upgrade pipx
- pipx ensurepath
- pipx install hatch --force
- - name: Run tests
- run: hatch run run-coverage
- env:
- HATCH_ENV: "test"
- working-directory: ./clients/python
- - name: "Install Airflow in editable mode with fab for webserver tests"
- run: pip install -e ".[fab]"
- - name: "Install Python client"
- run: pip install ./dist/apache_airflow_client-*.whl
- - name: "Initialize Airflow DB and start webserver"
- run: |
- airflow db init
- # Let scheduler runs a few loops and get all DAG files from example DAGs serialized to DB
- airflow scheduler --num-runs 100
- airflow users create --username admin --password admin --firstname Admin --lastname Admin \
- --role Admin --email admin@example.org
- killall python || true # just in case there is a webserver running in the background
- nohup airflow webserver --port 8080 &
- echo "Started webserver"
- env:
- AIRFLOW__API__AUTH_BACKENDS: airflow.api.auth.backend.session,airflow.api.auth.backend.basic_auth
- AIRFLOW__WEBSERVER__EXPOSE_CONFIG: "True"
- AIRFLOW__CORE__LOAD_EXAMPLES: "True"
- AIRFLOW_HOME: "${{ github.workspace }}/airflow_home"
- - name: "Waiting for the webserver to be available"
- run: |
- timeout 30 bash -c 'until nc -z $0 $1; do echo "sleeping"; sleep 1; done' localhost 8080
- sleep 5
- - name: "Run test python client"
- run: python ./clients/python/test_python_client.py
- env:
- FORCE_COLOR: "standard"
- - name: "Stop running webserver"
- run: killall python || true # just in case there is a webserver running in the background
- if: always()
- - name: "Upload python client packages"
- uses: actions/upload-artifact@v4
with:
- name: python-client-packages
- path: ./dist/apache_airflow_client-*
- retention-days: 7
- if-no-files-found: error
- - name: "Upload logs from failed tests"
- uses: actions/upload-artifact@v4
- if: failure()
+ use-uv: ${{ inputs.use-uv }}
+ id: breeze
+ - name: "Install pre-commit"
+ uses: ./.github/actions/install-pre-commit
+ id: pre-commit
with:
- name: python-client-failed-logs
- path: "${{ github.workspace }}/airflow_home/logs"
- retention-days: 7
+ python-version: ${{steps.breeze.outputs.host-python-version}}
# Those checks are run if no image needs to be built for checks. This is for simple changes that
# Do not touch any of the python code or any of the important files that might require building
@@ -206,6 +198,7 @@ jobs:
timeout-minutes: 30
name: "Static checks: basic checks only"
runs-on: ${{ fromJSON(inputs.runs-on-as-json-public) }}
+ needs: install-pre-commit
if: inputs.basic-checks-only == 'true'
steps:
- name: "Cleanup repo"
@@ -217,20 +210,10 @@ jobs:
persist-credentials: false
- name: "Cleanup docker"
run: ./scripts/ci/cleanup_docker.sh
- - name: "Setup python"
- uses: actions/setup-python@v5
- with:
- python-version: ${{ inputs.default-python-version }}
- cache: 'pip'
- cache-dependency-path: ./dev/breeze/pyproject.toml
- - name: "Setup python"
- uses: actions/setup-python@v5
- with:
- python-version: "${{ inputs.default-python-version }}"
- cache: 'pip'
- cache-dependency-path: ./dev/breeze/pyproject.toml
- name: "Install Breeze"
uses: ./.github/actions/breeze
+ with:
+ use-uv: ${{ inputs.use-uv }}
id: breeze
- name: "Install pre-commit"
uses: ./.github/actions/install-pre-commit
@@ -268,6 +251,7 @@ jobs:
timeout-minutes: 45
name: "Upgrade checks"
runs-on: ${{ fromJSON(inputs.runs-on-as-json-public) }}
+ needs: install-pre-commit
env:
PYTHON_MAJOR_MINOR_VERSION: "${{ inputs.default-python-version }}"
if: inputs.canary-run == 'true' && inputs.latest-versions-only != 'true'
@@ -281,12 +265,16 @@ jobs:
persist-credentials: false
- name: "Cleanup docker"
run: ./scripts/ci/cleanup_docker.sh
- # Install python from scratch. No cache used. We always want to have fresh version of everything
- - uses: actions/setup-python@v5
+ - name: "Install Breeze"
+ uses: ./.github/actions/breeze
+ with:
+ use-uv: ${{ inputs.use-uv }}
+ id: breeze
+ - name: "Install pre-commit"
+ uses: ./.github/actions/install-pre-commit
+ id: pre-commit
with:
- python-version: "${{ inputs.default-python-version }}"
- - name: "Install latest pre-commit"
- run: pip install pre-commit
+ python-version: ${{steps.breeze.outputs.host-python-version}}
- name: "Autoupdate all pre-commits"
run: pre-commit autoupdate
- name: "Run automated upgrade for black"
@@ -319,11 +307,12 @@ jobs:
run: >
pre-commit run
--all-files --show-diff-on-failure --color always --verbose
- --hook-stage manual update-installers || true
+ --hook-stage manual update-installers-and-pre-commit || true
if: always()
env:
UPGRADE_UV: "true"
UPGRADE_PIP: "false"
+ UPGRADE_PRE_COMMIT: "true"
- name: "Run automated upgrade for pip"
run: >
pre-commit run
@@ -340,11 +329,11 @@ jobs:
runs-on: ${{ fromJSON(inputs.runs-on-as-json-public) }}
env:
PYTHON_MAJOR_MINOR_VERSION: "${{ inputs.default-python-version }}"
- IMAGE_TAG: ${{ inputs.image-tag }}
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_USERNAME: ${{ github.actor }}
VERBOSE: "true"
+ if: inputs.canary-run == 'true'
steps:
- name: "Cleanup repo"
shell: bash
@@ -357,6 +346,8 @@ jobs:
run: ./scripts/ci/cleanup_docker.sh
- name: "Install Breeze"
uses: ./.github/actions/breeze
+ with:
+ use-uv: ${{ inputs.use-uv }}
- name: "Cleanup dist files"
run: rm -fv ./dist/*
- name: Setup git for tagging
@@ -366,18 +357,20 @@ jobs:
- name: Install twine
run: pip install twine
- name: "Check Airflow create minor branch command"
- run: breeze release-management create-minor-branch --version-branch 2-8 --answer yes
+ run: |
+ ./scripts/ci/testing/run_breeze_command_with_retries.sh \
+ release-management create-minor-branch --version-branch 2-8 --answer yes
- name: "Check Airflow RC process command"
- run: >
- breeze release-management start-rc-process --version 2.8.3rc1 --previous-version 2.8.0
- --sync-branch sync_v2_8_test --answer yes
+ run: |
+ ./scripts/ci/testing/run_breeze_command_with_retries.sh \
+ release-management start-rc-process --version 2.8.3rc1 --previous-version 2.8.0 --answer yes
- name: "Check Airflow release process command"
- run: >
- breeze release-management start-release --release-candidate 2.8.3rc1 --previous-release 2.8.0
- --answer yes
+ run: |
+ ./scripts/ci/testing/run_breeze_command_with_retries.sh \
+ release-management start-release --release-candidate 2.8.3rc1 --previous-release 2.8.0 --answer yes
- name: "Fetch all git tags"
run: git fetch --tags >/dev/null 2>&1 || true
- name: "Test airflow core issue generation automatically"
run: |
- breeze release-management generate-issue-content-core \
- --limit-pr-count 25 --latest --verbose
+ ./scripts/ci/testing/run_breeze_command_with_retries.sh \
+ release-management generate-issue-content-core --limit-pr-count 25 --latest --verbose
diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml
deleted file mode 100644
index 55e6c5d2018..00000000000
--- a/.github/workflows/build-images.yml
+++ /dev/null
@@ -1,259 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
----
-name: Build Images
-run-name: >
- Build images for ${{ github.event.pull_request.title }} ${{ github.event.pull_request._links.html.href }}
-on: # yamllint disable-line rule:truthy
- pull_request_target:
- branches:
- - main
- - v2-10-stable
- - v2-10-test
-permissions:
- # all other permissions are set to none
- contents: read
- pull-requests: read
- packages: read
-env:
- ANSWER: "yes"
- # You can override CONSTRAINTS_GITHUB_REPOSITORY by setting secret in your repo but by default the
- # Airflow one is going to be used
- CONSTRAINTS_GITHUB_REPOSITORY: >-
- ${{ secrets.CONSTRAINTS_GITHUB_REPOSITORY != '' &&
- secrets.CONSTRAINTS_GITHUB_REPOSITORY || 'apache/airflow' }}
- # This token is WRITE one - pull_request_target type of events always have the WRITE token
- DB_RESET: "true"
- GITHUB_REPOSITORY: ${{ github.repository }}
- GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- GITHUB_USERNAME: ${{ github.actor }}
- IMAGE_TAG: "${{ github.event.pull_request.head.sha || github.sha }}"
- INCLUDE_SUCCESS_OUTPUTS: "true"
- USE_SUDO: "true"
- VERBOSE: "true"
-
-concurrency:
- group: build-${{ github.event.pull_request.number || github.ref }}
- cancel-in-progress: true
-
-jobs:
- build-info:
- timeout-minutes: 10
- name: Build Info
- # At build-info stage we do not yet have outputs so we need to hard-code the runs-on to public runners
- runs-on: ["ubuntu-22.04"]
- env:
- TARGET_BRANCH: ${{ github.event.pull_request.base.ref }}
- outputs:
- image-tag: ${{ github.event.pull_request.head.sha || github.sha }}
- python-versions: ${{ steps.selective-checks.outputs.python-versions }}
- python-versions-list-as-string: ${{ steps.selective-checks.outputs.python-versions-list-as-string }}
- default-python-version: ${{ steps.selective-checks.outputs.default-python-version }}
- upgrade-to-newer-dependencies: ${{ steps.selective-checks.outputs.upgrade-to-newer-dependencies }}
- run-tests: ${{ steps.selective-checks.outputs.run-tests }}
- run-kubernetes-tests: ${{ steps.selective-checks.outputs.run-kubernetes-tests }}
- ci-image-build: ${{ steps.selective-checks.outputs.ci-image-build }}
- prod-image-build: ${{ steps.selective-checks.outputs.prod-image-build }}
- docker-cache: ${{ steps.selective-checks.outputs.docker-cache }}
- default-branch: ${{ steps.selective-checks.outputs.default-branch }}
- force-pip: ${{ steps.selective-checks.outputs.force-pip }}
- constraints-branch: ${{ steps.selective-checks.outputs.default-constraints-branch }}
- runs-on-as-json-default: ${{ steps.selective-checks.outputs.runs-on-as-json-default }}
- runs-on-as-json-public: ${{ steps.selective-checks.outputs.runs-on-as-json-public }}
- runs-on-as-json-self-hosted: ${{ steps.selective-checks.outputs.runs-on-as-json-self-hosted }}
- is-self-hosted-runner: ${{ steps.selective-checks.outputs.is-self-hosted-runner }}
- is-committer-build: ${{ steps.selective-checks.outputs.is-committer-build }}
- is-airflow-runner: ${{ steps.selective-checks.outputs.is-airflow-runner }}
- is-amd-runner: ${{ steps.selective-checks.outputs.is-amd-runner }}
- is-arm-runner: ${{ steps.selective-checks.outputs.is-arm-runner }}
- is-vm-runner: ${{ steps.selective-checks.outputs.is-vm-runner }}
- is-k8s-runner: ${{ steps.selective-checks.outputs.is-k8s-runner }}
- chicken-egg-providers: ${{ steps.selective-checks.outputs.chicken-egg-providers }}
- target-commit-sha: "${{steps.discover-pr-merge-commit.outputs.target-commit-sha ||
- github.event.pull_request.head.sha ||
- github.sha
- }}"
- if: github.repository == 'apache/airflow'
- steps:
- - name: Cleanup repo
- shell: bash
- run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
- - name: Discover PR merge commit
- id: discover-pr-merge-commit
- run: |
- # Sometimes target-commit-sha cannot be
- TARGET_COMMIT_SHA="$(gh api '${{ github.event.pull_request.url }}' --jq .merge_commit_sha)"
- if [[ ${TARGET_COMMIT_SHA} == "" ]]; then
- # Sometimes retrieving the merge commit SHA from PR fails. We retry it once. Otherwise we
- # fall-back to github.event.pull_request.head.sha
- echo
- echo "Could not retrieve merge commit SHA from PR, waiting for 3 seconds and retrying."
- echo
- sleep 3
- TARGET_COMMIT_SHA="$(gh api '${{ github.event.pull_request.url }}' --jq .merge_commit_sha)"
- if [[ ${TARGET_COMMIT_SHA} == "" ]]; then
- echo
- echo "Could not retrieve merge commit SHA from PR, falling back to PR head SHA."
- echo
- TARGET_COMMIT_SHA="${{ github.event.pull_request.head.sha }}"
- fi
- fi
- echo "TARGET_COMMIT_SHA=${TARGET_COMMIT_SHA}"
- echo "TARGET_COMMIT_SHA=${TARGET_COMMIT_SHA}" >> ${GITHUB_ENV}
- echo "target-commit-sha=${TARGET_COMMIT_SHA}" >> ${GITHUB_OUTPUT}
- if: github.event_name == 'pull_request_target'
- # The labels in the event aren't updated when re-triggering the job, So lets hit the API to get
- # up-to-date values
- - name: Get latest PR labels
- id: get-latest-pr-labels
- run: |
- echo -n "pull-request-labels=" >> ${GITHUB_OUTPUT}
- gh api graphql --paginate -F node_id=${{github.event.pull_request.node_id}} -f query='
- query($node_id: ID!, $endCursor: String) {
- node(id:$node_id) {
- ... on PullRequest {
- labels(first: 100, after: $endCursor) {
- nodes { name }
- pageInfo { hasNextPage endCursor }
- }
- }
- }
- }' --jq '.data.node.labels.nodes[]' | jq --slurp -c '[.[].name]' >> ${GITHUB_OUTPUT}
- if: github.event_name == 'pull_request_target'
- - uses: actions/checkout@v4
- with:
- ref: ${{ env.TARGET_COMMIT_SHA }}
- persist-credentials: false
- fetch-depth: 2
- ####################################################################################################
- # WE ONLY DO THAT CHECKOUT ABOVE TO RETRIEVE THE TARGET COMMIT AND IT'S PARENT. DO NOT RUN ANY CODE
- # RIGHT AFTER THAT AS WE ARE GOING TO RESTORE THE TARGET BRANCH CODE IN THE NEXT STEP.
- ####################################################################################################
- - name: Checkout target branch to use ci/scripts and breeze from there.
- uses: actions/checkout@v4
- with:
- ref: ${{ github.base_ref }}
- persist-credentials: false
- ####################################################################################################
- # HERE EVERYTHING IS PERFECTLY SAFE TO RUN. AT THIS POINT WE HAVE THE TARGET BRANCH CHECKED OUT
- # AND WE CAN RUN ANY CODE FROM IT. WE CAN RUN BREEZE COMMANDS, WE CAN RUN SCRIPTS, WE CAN RUN
- # COMPOSITE ACTIONS. WE CAN RUN ANYTHING THAT IS IN THE TARGET BRANCH AND THERE IS NO RISK THAT
- # CODE WILL BE RUN FROM THE PR.
- ####################################################################################################
- - name: Cleanup docker
- run: ./scripts/ci/cleanup_docker.sh
- - name: Setup python
- uses: actions/setup-python@v5
- with:
- python-version: "3.9"
- - name: Install Breeze
- uses: ./.github/actions/breeze
- ####################################################################################################
- # WE RUN SELECTIVE CHECKS HERE USING THE TARGET COMMIT AND ITS PARENT TO BE ABLE TO COMPARE THEM
- # AND SEE WHAT HAS CHANGED IN THE PR. THE CODE IS STILL RUN FROM THE TARGET BRANCH, SO IT IS SAFE
- # TO RUN IT, WE ONLY PASS TARGET_COMMIT_SHA SO THAT SELECTIVE CHECKS CAN SEE WHAT'S COMING IN THE PR
- ####################################################################################################
- - name: Selective checks
- id: selective-checks
- env:
- PR_LABELS: "${{ steps.get-latest-pr-labels.outputs.pull-request-labels }}"
- COMMIT_REF: "${{ env.TARGET_COMMIT_SHA }}"
- VERBOSE: "false"
- AIRFLOW_SOURCES_ROOT: "${{ github.workspace }}"
- run: breeze ci selective-check 2>> ${GITHUB_OUTPUT}
- - name: env
- run: printenv
- env:
- PR_LABELS: ${{ steps.get-latest-pr-labels.outputs.pull-request-labels }}
- GITHUB_CONTEXT: ${{ toJson(github) }}
-
-
- build-ci-images:
- name: Build CI images
- permissions:
- contents: read
- packages: write
- secrets: inherit
- needs: [build-info]
- uses: ./.github/workflows/ci-image-build.yml
- # Only run this it if the PR comes from fork, otherwise build will be done "in-PR-workflow"
- if: |
- needs.build-info.outputs.ci-image-build == 'true' &&
- github.event.pull_request.head.repo.full_name != 'apache/airflow'
- with:
- runs-on-as-json-public: ${{ needs.build-info.outputs.runs-on-as-json-public }}
- runs-on-as-json-self-hosted: ${{ needs.build-info.outputs.runs-on-as-json-self-hosted }}
- do-build: ${{ needs.build-info.outputs.ci-image-build }}
- target-commit-sha: ${{ needs.build-info.outputs.target-commit-sha }}
- pull-request-target: "true"
- is-committer-build: ${{ needs.build-info.outputs.is-committer-build }}
- push-image: "true"
- use-uv: ${{ needs.build-info.outputs.force-pip && 'false' || 'true' }}
- image-tag: ${{ needs.build-info.outputs.image-tag }}
- platform: "linux/amd64"
- python-versions: ${{ needs.build-info.outputs.python-versions }}
- branch: ${{ needs.build-info.outputs.default-branch }}
- constraints-branch: ${{ needs.build-info.outputs.constraints-branch }}
- upgrade-to-newer-dependencies: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
- docker-cache: ${{ needs.build-info.outputs.docker-cache }}
-
- generate-constraints:
- name: Generate constraints
- needs: [build-info, build-ci-images]
- uses: ./.github/workflows/generate-constraints.yml
- with:
- runs-on-as-json-public: ${{ needs.build-info.outputs.runs-on-as-json-public }}
- python-versions-list-as-string: ${{ needs.build-info.outputs.python-versions-list-as-string }}
- # For regular PRs we do not need "no providers" constraints - they are only needed in canary builds
- generate-no-providers-constraints: "false"
- image-tag: ${{ needs.build-info.outputs.image-tag }}
- chicken-egg-providers: ${{ needs.build-info.outputs.chicken-egg-providers }}
- debug-resources: ${{ needs.build-info.outputs.debug-resources }}
-
- build-prod-images:
- name: Build PROD images
- permissions:
- contents: read
- packages: write
- secrets: inherit
- needs: [build-info, generate-constraints]
- uses: ./.github/workflows/prod-image-build.yml
- # Only run this it if the PR comes from fork, otherwise build will be done "in-PR-workflow"
- if: |
- needs.build-info.outputs.prod-image-build == 'true' &&
- github.event.pull_request.head.repo.full_name != 'apache/airflow'
- with:
- runs-on-as-json-public: ${{ needs.build-info.outputs.runs-on-as-json-public }}
- build-type: "Regular"
- do-build: ${{ needs.build-info.outputs.ci-image-build }}
- upload-package-artifact: "true"
- target-commit-sha: ${{ needs.build-info.outputs.target-commit-sha }}
- pull-request-target: "true"
- is-committer-build: ${{ needs.build-info.outputs.is-committer-build }}
- push-image: "true"
- use-uv: ${{ needs.build-info.outputs.force-pip && 'false' || 'true' }}
- image-tag: ${{ needs.build-info.outputs.image-tag }}
- platform: linux/amd64
- python-versions: ${{ needs.build-info.outputs.python-versions }}
- default-python-version: ${{ needs.build-info.outputs.default-python-version }}
- branch: ${{ needs.build-info.outputs.default-branch }}
- constraints-branch: ${{ needs.build-info.outputs.constraints-branch }}
- build-provider-packages: ${{ needs.build-info.outputs.default-branch == 'main' }}
- upgrade-to-newer-dependencies: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
- chicken-egg-providers: ${{ needs.build-info.outputs.chicken-egg-providers }}
- docker-cache: ${{ needs.build-info.outputs.docker-cache }}
diff --git a/.github/workflows/ci-image-build.yml b/.github/workflows/ci-image-build.yml
index 1c4b31b55a6..d15c297d82a 100644
--- a/.github/workflows/ci-image-build.yml
+++ b/.github/workflows/ci-image-build.yml
@@ -28,13 +28,6 @@ on: # yamllint disable-line rule:truthy
description: "The array of labels (in json form) determining self-hosted runners."
required: true
type: string
- do-build:
- description: >
- Whether to actually do the build (true/false). If set to false, the build is done
- already in pull-request-target workflow, so we skip it here.
- required: false
- default: "true"
- type: string
target-commit-sha:
description: "The commit SHA to checkout for the build"
required: false
@@ -59,6 +52,14 @@ on: # yamllint disable-line rule:truthy
required: false
default: "true"
type: string
+ upload-image-artifact:
+ description: "Whether to upload docker image artifact"
+ required: true
+ type: string
+ upload-mount-cache-artifact:
+ description: "Whether to upload mount-cache artifact"
+ required: true
+ type: string
debian-version:
description: "Base Debian distribution to use for the build (bookworm)"
type: string
@@ -71,10 +72,6 @@ on: # yamllint disable-line rule:truthy
description: "Whether to use uv to build the image (true/false)"
required: true
type: string
- image-tag:
- description: "Tag to set for the image"
- required: true
- type: string
python-versions:
description: "JSON-formatted array of Python versions to build images from"
required: true
@@ -95,25 +92,18 @@ on: # yamllint disable-line rule:truthy
description: "Docker cache specification to build the image (registry, local, disabled)."
required: true
type: string
+ disable-airflow-repo-cache:
+ description: "Disable airflow repo cache read from main."
+ required: true
+ type: string
jobs:
build-ci-images:
strategy:
fail-fast: true
matrix:
- # yamllint disable-line rule:line-length
- python-version: ${{ inputs.do-build == 'true' && fromJSON(inputs.python-versions) || fromJSON('[""]') }}
+ python-version: ${{ fromJSON(inputs.python-versions) || fromJSON('[""]') }}
timeout-minutes: 110
- name: "\
-${{ inputs.do-build == 'true' && 'Build' || 'Skip building' }} \
-CI ${{ inputs.platform }} image\
-${{ matrix.python-version }}${{ inputs.do-build == 'true' && ':' || '' }}\
-${{ inputs.do-build == 'true' && inputs.image-tag || '' }}"
- # The ARM images need to be built using self-hosted runners as ARM macos public runners
- # do not yet allow us to run docker effectively and fast.
- # https://github.com/actions/runner-images/issues/9254#issuecomment-1917916016
- # https://github.com/abiosoft/colima/issues/970
- # https://github.com/actions/runner/issues/1456
- # See https://github.com/apache/airflow/pull/38640
+ name: "Build CI ${{ inputs.platform }} image ${{ matrix.python-version }}"
# NOTE!!!!! This has to be put in one line for runs-on to recognize the "fromJSON" properly !!!!
# adding space before (with >) apparently turns the `runs-on` processed line into a string "Array"
# instead of an array of strings.
@@ -121,56 +111,54 @@ ${{ inputs.do-build == 'true' && inputs.image-tag || '' }}"
runs-on: ${{ (inputs.platform == 'linux/amd64') && fromJSON(inputs.runs-on-as-json-public) || fromJSON(inputs.runs-on-as-json-self-hosted) }}
env:
BACKEND: sqlite
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
DEFAULT_BRANCH: ${{ inputs.branch }}
DEFAULT_CONSTRAINTS_BRANCH: ${{ inputs.constraints-branch }}
VERSION_SUFFIX_FOR_PYPI: "dev0"
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_USERNAME: ${{ github.actor }}
- USE_UV: ${{ inputs.use-uv }}
VERBOSE: "true"
steps:
- name: "Cleanup repo"
shell: bash
run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
- if: inputs.do-build == 'true'
- name: "Checkout target branch"
uses: actions/checkout@v4
with:
persist-credentials: false
- - name: "Checkout target commit"
- uses: ./.github/actions/checkout_target_commit
- if: inputs.do-build == 'true'
- with:
- target-commit-sha: ${{ inputs.target-commit-sha }}
- pull-request-target: ${{ inputs.pull-request-target }}
- is-committer-build: ${{ inputs.is-committer-build }}
- name: "Cleanup docker"
run: ./scripts/ci/cleanup_docker.sh
- if: inputs.do-build == 'true'
- name: "Install Breeze"
uses: ./.github/actions/breeze
- if: inputs.do-build == 'true'
- - name: "Regenerate dependencies in case they were modified manually so that we can build an image"
- shell: bash
- run: |
- pip install rich>=12.4.4 pyyaml
- python scripts/ci/pre_commit/update_providers_dependencies.py
- if: inputs.do-build == 'true' && inputs.upgrade-to-newer-dependencies != 'false'
- - name: "Start ARM instance"
- run: ./scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh
- if: inputs.do-build == 'true' && inputs.platform == 'linux/arm64'
- - name: Login to ghcr.io
- run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
- if: inputs.do-build == 'true'
+ with:
+ use-uv: ${{ inputs.use-uv }}
+ - name: "Restore ci-cache mount image ${{ inputs.platform }}:${{ env.PYTHON_MAJOR_MINOR_VERSION }}"
+ uses: apache/infrastructure-actions/stash/restore@c94b890bbedc2fc61466d28e6bd9966bc6c6643c
+ with:
+ key: "ci-cache-mount-save-v2-${{ inputs.platform }}-${{ env.PYTHON_MAJOR_MINOR_VERSION }}"
+ path: "/tmp/"
+ id: restore-cache-mount
+ - name: "Import mount-cache ${{ inputs.platform }}:${{ env.PYTHON_MAJOR_MINOR_VERSION }}"
+ env:
+ PYTHON_MAJOR_MINOR_VERSION: ${{ env.PYTHON_MAJOR_MINOR_VERSION }}
+ run: >
+ breeze ci-image import-mount-cache
+ --cache-file /tmp/ci-cache-mount-save-v2-${PYTHON_MAJOR_MINOR_VERSION}.tar.gz
+ if: steps.restore-cache-mount.outputs.stash-hit == 'true'
+ - name: "Login to ghcr.io"
+ env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ ACTOR: ${{ github.actor }}
+ run: echo "${GITHUB_TOKEN}" | docker login ghcr.io -u ${ACTOR} --password-stdin
- name: >
Build ${{ inputs.push-image == 'true' && ' & push ' || '' }}
- ${{ inputs.platform }}:${{ matrix.python-version }}:${{ inputs.image-tag }}
+ ${{ inputs.platform }}:${{ env.PYTHON_MAJOR_MINOR_VERSION }} image
run: >
- breeze ci-image build --builder airflow_cache --tag-as-latest --image-tag "${{ inputs.image-tag }}"
- --python "${{ matrix.python-version }}" --platform "${{ inputs.platform }}"
+ breeze ci-image build --platform "${PLATFORM}"
env:
DOCKER_CACHE: ${{ inputs.docker-cache }}
+ DISABLE_AIRFLOW_REPO_CACHE: ${{ inputs.disable-airflow-repo-cache }}
INSTALL_MYSQL_CLIENT_TYPE: ${{ inputs.install-mysql-client-type }}
UPGRADE_TO_NEWER_DEPENDENCIES: ${{ inputs.upgrade-to-newer-dependencies }}
# You can override CONSTRAINTS_GITHUB_REPOSITORY by setting secret in your repo but by default the
@@ -184,7 +172,32 @@ ${{ inputs.do-build == 'true' && inputs.image-tag || '' }}"
GITHUB_USERNAME: ${{ github.actor }}
PUSH: ${{ inputs.push-image }}
VERBOSE: "true"
- if: inputs.do-build == 'true'
- - name: "Stop ARM instance"
- run: ./scripts/ci/images/ci_stop_arm_instance.sh
- if: always() && inputs.do-build == 'true' && inputs.platform == 'linux/arm64'
+ PLATFORM: ${{ inputs.platform }}
+ - name: "Export CI docker image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}"
+ env:
+ PLATFORM: ${{ inputs.platform }}
+ run: breeze ci-image save --platform "${PLATFORM}"
+ if: inputs.upload-image-artifact == 'true'
+ - name: "Stash CI docker image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}"
+ uses: apache/infrastructure-actions/stash/save@c94b890bbedc2fc61466d28e6bd9966bc6c6643c
+ with:
+ key: ci-image-save-${{ inputs.platform }}-${{ env.PYTHON_MAJOR_MINOR_VERSION }}
+ path: "/tmp/ci-image-save-*-${{ env.PYTHON_MAJOR_MINOR_VERSION }}.tar"
+ if-no-files-found: 'error'
+ retention-days: '2'
+ if: inputs.upload-image-artifact == 'true'
+ - name: "Export mount cache ${{ inputs.platform }}:${{ env.PYTHON_MAJOR_MINOR_VERSION }}"
+ env:
+ PYTHON_MAJOR_MINOR_VERSION: ${{ env.PYTHON_MAJOR_MINOR_VERSION }}
+ run: >
+ breeze ci-image export-mount-cache
+ --cache-file /tmp/ci-cache-mount-save-v2-${PYTHON_MAJOR_MINOR_VERSION}.tar.gz
+ if: inputs.upload-mount-cache-artifact == 'true'
+ - name: "Stash cache mount ${{ inputs.platform }}:${{ env.PYTHON_MAJOR_MINOR_VERSION }}"
+ uses: apache/infrastructure-actions/stash/save@c94b890bbedc2fc61466d28e6bd9966bc6c6643c
+ with:
+ key: "ci-cache-mount-save-v2-${{ inputs.platform }}-${{ env.PYTHON_MAJOR_MINOR_VERSION }}"
+ path: "/tmp/ci-cache-mount-save-v2-${{ env.PYTHON_MAJOR_MINOR_VERSION }}.tar.gz"
+ if-no-files-found: 'error'
+ retention-days: 2
+ if: inputs.upload-mount-cache-artifact == 'true'
diff --git a/.github/workflows/static-checks-mypy-docs.yml b/.github/workflows/ci-image-checks.yml
similarity index 64%
rename from .github/workflows/static-checks-mypy-docs.yml
rename to .github/workflows/ci-image-checks.yml
index be2c4f8e286..21c857e7bd7 100644
--- a/.github/workflows/static-checks-mypy-docs.yml
+++ b/.github/workflows/ci-image-checks.yml
@@ -16,7 +16,7 @@
# under the License.
#
---
-name: Static checks, mypy, docs
+name: CI Image Checks
on: # yamllint disable-line rule:truthy
workflow_call:
inputs:
@@ -28,10 +28,6 @@ on: # yamllint disable-line rule:truthy
description: "The array of labels (in json form) determining the labels used for docs build."
required: true
type: string
- image-tag:
- description: "Tag to set for the image"
- required: true
- type: string
needs-mypy:
description: "Whether to run mypy checks (true/false)"
required: true
@@ -96,15 +92,76 @@ on: # yamllint disable-line rule:truthy
description: "Whether to build docs (true/false)"
required: true
type: string
+ needs-api-codegen:
+ description: "Whether to run API codegen (true/false)"
+ required: true
+ type: string
+ default-postgres-version:
+ description: "The default version of the postgres to use"
+ required: true
+ type: string
+ run-coverage:
+ description: "Whether to run coverage or not (true/false)"
+ required: true
+ type: string
+ use-uv:
+ description: "Whether to use uv to build the image (true/false)"
+ required: true
+ type: string
+
jobs:
+ install-pre-commit:
+ timeout-minutes: 5
+ name: "Install pre-commit for cache (only canary runs)"
+ runs-on: ${{ fromJSON(inputs.runs-on-as-json-default) }}
+ env:
+ PYTHON_MAJOR_MINOR_VERSION: "${{ inputs.default-python-version }}"
+ if: inputs.basic-checks-only == 'false'
+ steps:
+ - name: "Cleanup repo"
+ shell: bash
+ run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ if: inputs.canary-run == 'true'
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ uses: actions/checkout@v4
+ with:
+ persist-credentials: false
+ if: inputs.canary-run == 'true'
+ - name: "Install Breeze"
+ uses: ./.github/actions/breeze
+ with:
+ use-uv: ${{ inputs.use-uv }}
+ id: breeze
+ if: inputs.canary-run == 'true'
+ - name: "Install pre-commit"
+ uses: ./.github/actions/install-pre-commit
+ id: pre-commit
+ with:
+ python-version: ${{steps.breeze.outputs.host-python-version}}
+ if: inputs.canary-run == 'true'
+ - name: "Prepare .tar file from pre-commit cache"
+ run: |
+ tar -C ~ -czf /tmp/cache-pre-commit.tar.gz .cache/pre-commit .cache/uv
+ shell: bash
+ if: inputs.canary-run == 'true'
+ - name: "Save pre-commit cache"
+ uses: apache/infrastructure-actions/stash/save@c94b890bbedc2fc61466d28e6bd9966bc6c6643c
+ with:
+ # yamllint disable rule:line-length
+ key: cache-pre-commit-v4-${{ steps.breeze.outputs.host-python-version }}-${{ hashFiles('.pre-commit-config.yaml') }}
+ path: /tmp/cache-pre-commit.tar.gz
+ if-no-files-found: 'error'
+ retention-days: '2'
+ if: inputs.canary-run == 'true'
+
static-checks:
timeout-minutes: 45
name: "Static checks"
runs-on: ${{ fromJSON(inputs.runs-on-as-json-default) }}
+ needs: install-pre-commit
env:
PYTHON_MAJOR_MINOR_VERSION: "${{ inputs.default-python-version }}"
UPGRADE_TO_NEWER_DEPENDENCIES: "${{ inputs.upgrade-to-newer-dependencies }}"
- IMAGE_TAG: ${{ inputs.image-tag }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
if: inputs.basic-checks-only == 'false' && inputs.latest-versions-only != 'true'
steps:
@@ -115,16 +172,12 @@ jobs:
uses: actions/checkout@v4
with:
persist-credentials: false
- - name: "Setup python"
- uses: actions/setup-python@v5
- with:
- python-version: ${{ inputs.default-python-version }}
- cache: 'pip'
- cache-dependency-path: ./dev/breeze/pyproject.toml
- - name: "Cleanup docker"
- run: ./scripts/ci/cleanup_docker.sh
- - name: "Prepare breeze & CI image: ${{ inputs.default-python-version}}:${{ inputs.image-tag }}"
+ - name: "Prepare breeze & CI image: ${{ inputs.default-python-version }}"
uses: ./.github/actions/prepare_breeze_and_image
+ with:
+ platform: "linux/amd64"
+ python: ${{ inputs.default-python-version }}
+ use-uv: ${{ inputs.use-uv }}
id: breeze
- name: "Install pre-commit"
uses: ./.github/actions/install-pre-commit
@@ -145,6 +198,7 @@ jobs:
timeout-minutes: 45
name: "MyPy checks"
runs-on: ${{ fromJSON(inputs.runs-on-as-json-default) }}
+ needs: install-pre-commit
if: inputs.needs-mypy == 'true'
strategy:
fail-fast: false
@@ -152,7 +206,6 @@ jobs:
mypy-check: ${{ fromJSON(inputs.mypy-checks) }}
env:
PYTHON_MAJOR_MINOR_VERSION: "${{inputs.default-python-version}}"
- IMAGE_TAG: "${{ inputs.image-tag }}"
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- name: "Cleanup repo"
@@ -162,10 +215,12 @@ jobs:
uses: actions/checkout@v4
with:
persist-credentials: false
- - name: "Cleanup docker"
- run: ./scripts/ci/cleanup_docker.sh
- - name: "Prepare breeze & CI image: ${{ inputs.default-python-version }}:${{ inputs.image-tag }}"
+ - name: "Prepare breeze & CI image: ${{ inputs.default-python-version }}"
uses: ./.github/actions/prepare_breeze_and_image
+ with:
+ platform: "linux/amd64"
+ python: ${{ inputs.default-python-version }}
+ use-uv: ${{ inputs.use-uv }}
id: breeze
- name: "Install pre-commit"
uses: ./.github/actions/install-pre-commit
@@ -173,7 +228,7 @@ jobs:
with:
python-version: ${{steps.breeze.outputs.host-python-version}}
- name: "MyPy checks for ${{ matrix.mypy-check }}"
- run: pre-commit run --color always --verbose --hook-stage manual ${{matrix.mypy-check}} --all-files
+ run: pre-commit run --color always --verbose --hook-stage manual "$MYPY_CHECK" --all-files
env:
VERBOSE: "false"
COLUMNS: "250"
@@ -181,6 +236,7 @@ jobs:
DEFAULT_BRANCH: ${{ inputs.branch }}
RUFF_FORMAT: "github"
INCLUDE_MYPY_VOLUME: "false"
+ MYPY_CHECK: ${{ matrix.mypy-check }}
build-docs:
timeout-minutes: 150
@@ -195,7 +251,6 @@ jobs:
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_USERNAME: ${{ github.actor }}
- IMAGE_TAG: "${{ inputs.image-tag }}"
INCLUDE_NOT_READY_PROVIDERS: "true"
INCLUDE_SUCCESS_OUTPUTS: "${{ inputs.include-success-outputs }}"
PYTHON_MAJOR_MINOR_VERSION: "${{ inputs.default-python-version }}"
@@ -208,28 +263,39 @@ jobs:
uses: actions/checkout@v4
with:
persist-credentials: false
- - name: "Cleanup docker"
- run: ./scripts/ci/cleanup_docker.sh
- - name: "Prepare breeze & CI image: ${{ inputs.default-python-version }}:${{ inputs.image-tag }}"
+ - name: "Prepare breeze & CI image: ${{ inputs.default-python-version }}"
uses: ./.github/actions/prepare_breeze_and_image
- - uses: actions/cache@v4
- id: cache-doc-inventories
+ with:
+ platform: "linux/amd64"
+ python: ${{ inputs.default-python-version }}
+ use-uv: ${{ inputs.use-uv }}
+ - name: "Restore docs inventory cache"
+ uses: apache/infrastructure-actions/stash/restore@c94b890bbedc2fc61466d28e6bd9966bc6c6643c
with:
path: ./docs/_inventory_cache/
- key: docs-inventory-${{ hashFiles('pyproject.toml;') }}
- restore-keys: |
- docs-inventory-${{ hashFiles('pyproject.toml;') }}
- docs-inventory-
+ # TODO(potiuk): do better with determining the key
+ key: cache-docs-inventory-v1-${{ hashFiles('pyproject.toml') }}
+ id: restore-docs-inventory-cache
- name: "Building docs with ${{ matrix.flag }} flag"
+ env:
+ DOCS_LIST_AS_STRING: ${{ inputs.docs-list-as-string }}
run: >
- breeze build-docs ${{ inputs.docs-list-as-string }} ${{ matrix.flag }}
+ breeze build-docs ${DOCS_LIST_AS_STRING} ${{ matrix.flag }}
+ - name: "Save docs inventory cache"
+ uses: apache/infrastructure-actions/stash/save@c94b890bbedc2fc61466d28e6bd9966bc6c6643c
+ with:
+ path: ./docs/_inventory_cache/
+ key: cache-docs-inventory-v1-${{ hashFiles('pyproject.toml') }}
+ if-no-files-found: 'error'
+ retention-days: '2'
+ if: steps.restore-docs-inventory-cache != 'true'
- name: "Upload build docs"
uses: actions/upload-artifact@v4
with:
name: airflow-docs
path: './docs/_build'
- retention-days: 7
- if-no-files-found: error
+ retention-days: '7'
+ if-no-files-found: 'error'
if: matrix.flag == '--docs-only'
publish-docs:
@@ -241,7 +307,6 @@ jobs:
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_USERNAME: ${{ github.actor }}
- IMAGE_TAG: "${{ inputs.image-tag }}"
INCLUDE_NOT_READY_PROVIDERS: "true"
INCLUDE_SUCCESS_OUTPUTS: "${{ inputs.include-success-outputs }}"
PYTHON_MAJOR_MINOR_VERSION: "${{ inputs.default-python-version }}"
@@ -270,12 +335,18 @@ jobs:
run: >
git clone https://github.com/apache/airflow-site.git /mnt/airflow-site/airflow-site &&
echo "AIRFLOW_SITE_DIRECTORY=/mnt/airflow-site/airflow-site" >> "$GITHUB_ENV"
- - name: "Prepare breeze & CI image: ${{ inputs.default-python-version }}:${{ inputs.image-tag }}"
+ - name: "Prepare breeze & CI image: ${{ inputs.default-python-version }}"
uses: ./.github/actions/prepare_breeze_and_image
+ with:
+ platform: "linux/amd64"
+ python: ${{ inputs.default-python-version }}
+ use-uv: ${{ inputs.use-uv }}
- name: "Publish docs"
+ env:
+ DOCS_LIST_AS_STRING: ${{ inputs.docs-list-as-string }}
run: >
breeze release-management publish-docs --override-versioned --run-in-parallel
- ${{ inputs.docs-list-as-string }}
+ ${DOCS_LIST_AS_STRING}
- name: Check disk space available
run: df -h
- name: "Generate back references for providers"
@@ -304,3 +375,53 @@ jobs:
- name: "Upload documentation to AWS S3"
if: inputs.branch == 'main'
run: aws s3 sync --delete ./docs/_build s3://apache-airflow-docs
+
+ test-python-api-client:
+ timeout-minutes: 60
+ name: "Test Python API client"
+ runs-on: ${{ fromJSON(inputs.runs-on-as-json-default) }}
+ if: inputs.needs-api-codegen == 'true'
+ env:
+ BACKEND: "postgres"
+ BACKEND_VERSION: "${{ inputs.default-postgres-version }}"
+ DEBUG_RESOURCES: "${{ inputs.debug-resources }}"
+ ENABLE_COVERAGE: "${{ inputs.run-coverage }}"
+ GITHUB_REPOSITORY: ${{ github.repository }}
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ GITHUB_USERNAME: ${{ github.actor }}
+ JOB_ID: "python-api-client-tests"
+ PYTHON_MAJOR_MINOR_VERSION: "${{ inputs.default-python-version }}"
+ VERBOSE: "true"
+ steps:
+ - name: "Cleanup repo"
+ shell: bash
+ run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ uses: actions/checkout@v4
+ with:
+ fetch-depth: 2
+ persist-credentials: false
+ - name: "Cleanup docker"
+ run: ./scripts/ci/cleanup_docker.sh
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ uses: actions/checkout@v4
+ with:
+ repository: "apache/airflow-client-python"
+ fetch-depth: 1
+ persist-credentials: false
+ path: ./airflow-client-python
+ - name: "Prepare breeze & CI image: ${{ inputs.default-python-version }}"
+ uses: ./.github/actions/prepare_breeze_and_image
+ with:
+ platform: "linux/amd64"
+ python: ${{ inputs.default-python-version }}
+ use-uv: ${{ inputs.use-uv }}
+ - name: "Generate airflow python client"
+ run: >
+ breeze release-management prepare-python-client --package-format both
+ --version-suffix-for-pypi dev0 --python-client-repo ./airflow-client-python
+ - name: "Show diff"
+ run: git diff --color HEAD
+ working-directory: ./airflow-client-python
+ - name: "Python API client tests"
+ run: breeze testing python-api-client-tests
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 69272f7b901..d820bd4c6ec 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -21,24 +21,24 @@ on: # yamllint disable-line rule:truthy
schedule:
- cron: '28 1,7,13,19 * * *'
push:
- branches: ['v[0-9]+-[0-9]+-test']
+ branches:
+ - v[0-9]+-[0-9]+-test
+ - providers-[a-z]+-?[a-z]*/v[0-9]+-[0-9]+
pull_request:
- branches: ['main', 'v[0-9]+-[0-9]+-test', 'v[0-9]+-[0-9]+-stable']
+ branches:
+ - main
+ - v[0-9]+-[0-9]+-test
+ - v[0-9]+-[0-9]+-stable
+ - providers-[a-z]+-?[a-z]*/v[0-9]+-[0-9]+
workflow_dispatch:
permissions:
- # All other permissions are set to none
+ # All other permissions are set to none by default
contents: read
- # Technically read access while waiting for images should be more than enough. However,
- # there is a bug in GitHub Actions/Packages and in case private repositories are used, you get a permission
- # denied error when attempting to just pull private image, changing the token permission to write solves the
- # issue. This is not dangerous, because if it is for "apache/airflow", only maintainers can push ci.yml
- # changes. If it is for a fork, then the token is read-only anyway.
- packages: write
env:
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_USERNAME: ${{ github.actor }}
- IMAGE_TAG: "${{ github.event.pull_request.head.sha || github.sha }}"
+ SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
VERBOSE: "true"
concurrency:
@@ -54,81 +54,89 @@ jobs:
env:
GITHUB_CONTEXT: ${{ toJson(github) }}
outputs:
- image-tag: ${{ github.event.pull_request.head.sha || github.sha }}
- docker-cache: ${{ steps.selective-checks.outputs.docker-cache }}
- affected-providers-list-as-string: >-
- ${{ steps.selective-checks.outputs.affected-providers-list-as-string }}
- upgrade-to-newer-dependencies: ${{ steps.selective-checks.outputs.upgrade-to-newer-dependencies }}
- python-versions: ${{ steps.selective-checks.outputs.python-versions }}
- python-versions-list-as-string: ${{ steps.selective-checks.outputs.python-versions-list-as-string }}
all-python-versions-list-as-string: >-
${{ steps.selective-checks.outputs.all-python-versions-list-as-string }}
- default-python-version: ${{ steps.selective-checks.outputs.default-python-version }}
- kubernetes-versions-list-as-string: >-
- ${{ steps.selective-checks.outputs.kubernetes-versions-list-as-string }}
- kubernetes-combos-list-as-string: >-
- ${{ steps.selective-checks.outputs.kubernetes-combos-list-as-string }}
- default-kubernetes-version: ${{ steps.selective-checks.outputs.default-kubernetes-version }}
- postgres-versions: ${{ steps.selective-checks.outputs.postgres-versions }}
- default-postgres-version: ${{ steps.selective-checks.outputs.default-postgres-version }}
- mysql-versions: ${{ steps.selective-checks.outputs.mysql-versions }}
- default-mysql-version: ${{ steps.selective-checks.outputs.default-mysql-version }}
- default-helm-version: ${{ steps.selective-checks.outputs.default-helm-version }}
- default-kind-version: ${{ steps.selective-checks.outputs.default-kind-version }}
- force-pip: ${{ steps.selective-checks.outputs.force-pip }}
- full-tests-needed: ${{ steps.selective-checks.outputs.full-tests-needed }}
- parallel-test-types-list-as-string: >-
- ${{ steps.selective-checks.outputs.parallel-test-types-list-as-string }}
- providers-test-types-list-as-string: >-
- ${{ steps.selective-checks.outputs.providers-test-types-list-as-string }}
- separate-test-types-list-as-string: >-
- ${{ steps.selective-checks.outputs.separate-test-types-list-as-string }}
- include-success-outputs: ${{ steps.selective-checks.outputs.include-success-outputs }}
- postgres-exclude: ${{ steps.selective-checks.outputs.postgres-exclude }}
- mysql-exclude: ${{ steps.selective-checks.outputs.mysql-exclude }}
- sqlite-exclude: ${{ steps.selective-checks.outputs.sqlite-exclude }}
- skip-provider-tests: ${{ steps.selective-checks.outputs.skip-provider-tests }}
- run-tests: ${{ steps.selective-checks.outputs.run-tests }}
- run-amazon-tests: ${{ steps.selective-checks.outputs.run-amazon-tests }}
- run-www-tests: ${{ steps.selective-checks.outputs.run-www-tests }}
- run-kubernetes-tests: ${{ steps.selective-checks.outputs.run-kubernetes-tests }}
basic-checks-only: ${{ steps.selective-checks.outputs.basic-checks-only }}
+ canary-run: ${{ steps.source-run-info.outputs.canary-run }}
+ chicken-egg-providers: ${{ steps.selective-checks.outputs.chicken-egg-providers }}
ci-image-build: ${{ steps.selective-checks.outputs.ci-image-build }}
- prod-image-build: ${{ steps.selective-checks.outputs.prod-image-build }}
- docs-build: ${{ steps.selective-checks.outputs.docs-build }}
- mypy-checks: ${{ steps.selective-checks.outputs.mypy-checks }}
- needs-mypy: ${{ steps.selective-checks.outputs.needs-mypy }}
- needs-helm-tests: ${{ steps.selective-checks.outputs.needs-helm-tests }}
- needs-api-tests: ${{ steps.selective-checks.outputs.needs-api-tests }}
- needs-api-codegen: ${{ steps.selective-checks.outputs.needs-api-codegen }}
+ core-test-types-list-as-string: >-
+ ${{ steps.selective-checks.outputs.core-test-types-list-as-string }}
+ debug-resources: ${{ steps.selective-checks.outputs.debug-resources }}
default-branch: ${{ steps.selective-checks.outputs.default-branch }}
default-constraints-branch: ${{ steps.selective-checks.outputs.default-constraints-branch }}
+ default-helm-version: ${{ steps.selective-checks.outputs.default-helm-version }}
+ default-kind-version: ${{ steps.selective-checks.outputs.default-kind-version }}
+ default-kubernetes-version: ${{ steps.selective-checks.outputs.default-kubernetes-version }}
+ default-mysql-version: ${{ steps.selective-checks.outputs.default-mysql-version }}
+ default-postgres-version: ${{ steps.selective-checks.outputs.default-postgres-version }}
+ default-python-version: ${{ steps.selective-checks.outputs.default-python-version }}
+ disable-airflow-repo-cache: ${{ steps.selective-checks.outputs.disable-airflow-repo-cache }}
+ docker-cache: ${{ steps.selective-checks.outputs.docker-cache }}
+ docs-build: ${{ steps.selective-checks.outputs.docs-build }}
docs-list-as-string: ${{ steps.selective-checks.outputs.docs-list-as-string }}
- skip-pre-commits: ${{ steps.selective-checks.outputs.skip-pre-commits }}
- providers-compatibility-checks: ${{ steps.selective-checks.outputs.providers-compatibility-checks }}
+ excluded-providers-as-string: ${{ steps.selective-checks.outputs.excluded-providers-as-string }}
+ force-pip: ${{ steps.selective-checks.outputs.force-pip }}
+ full-tests-needed: ${{ steps.selective-checks.outputs.full-tests-needed }}
+ has-migrations: ${{ steps.selective-checks.outputs.has-migrations }}
helm-test-packages: ${{ steps.selective-checks.outputs.helm-test-packages }}
- debug-resources: ${{ steps.selective-checks.outputs.debug-resources }}
- runs-on-as-json-default: ${{ steps.selective-checks.outputs.runs-on-as-json-default }}
- runs-on-as-json-docs-build: ${{ steps.selective-checks.outputs.runs-on-as-json-docs-build }}
- runs-on-as-json-public: ${{ steps.selective-checks.outputs.runs-on-as-json-public }}
- runs-on-as-json-self-hosted: ${{ steps.selective-checks.outputs.runs-on-as-json-self-hosted }}
- runs-on-as-json-self-hosted-asf: ${{ steps.selective-checks.outputs.runs-on-as-json-self-hosted-asf }}
- is-self-hosted-runner: ${{ steps.selective-checks.outputs.is-self-hosted-runner }}
+ include-success-outputs: ${{ steps.selective-checks.outputs.include-success-outputs }}
+ individual-providers-test-types-list-as-string: >-
+ ${{ steps.selective-checks.outputs.individual-providers-test-types-list-as-string }}
is-airflow-runner: ${{ steps.selective-checks.outputs.is-airflow-runner }}
is-amd-runner: ${{ steps.selective-checks.outputs.is-amd-runner }}
is-arm-runner: ${{ steps.selective-checks.outputs.is-arm-runner }}
- is-vm-runner: ${{ steps.selective-checks.outputs.is-vm-runner }}
is-k8s-runner: ${{ steps.selective-checks.outputs.is-k8s-runner }}
+ is-self-hosted-runner: ${{ steps.selective-checks.outputs.is-self-hosted-runner }}
+ is-vm-runner: ${{ steps.selective-checks.outputs.is-vm-runner }}
+ kubernetes-combos: ${{ steps.selective-checks.outputs.kubernetes-combos }}
+ kubernetes-combos-list-as-string: >-
+ ${{ steps.selective-checks.outputs.kubernetes-combos-list-as-string }}
+ kubernetes-versions-list-as-string: >-
+ ${{ steps.selective-checks.outputs.kubernetes-versions-list-as-string }}
latest-versions-only: ${{ steps.selective-checks.outputs.latest-versions-only }}
- chicken-egg-providers: ${{ steps.selective-checks.outputs.chicken-egg-providers }}
- has-migrations: ${{ steps.selective-checks.outputs.has-migrations }}
- source-head-repo: ${{ steps.source-run-info.outputs.source-head-repo }}
+ mypy-checks: ${{ steps.selective-checks.outputs.mypy-checks }}
+ mysql-exclude: ${{ steps.selective-checks.outputs.mysql-exclude }}
+ mysql-versions: ${{ steps.selective-checks.outputs.mysql-versions }}
+ needs-api-codegen: ${{ steps.selective-checks.outputs.needs-api-codegen }}
+ needs-api-tests: ${{ steps.selective-checks.outputs.needs-api-tests }}
+ needs-helm-tests: ${{ steps.selective-checks.outputs.needs-helm-tests }}
+ needs-mypy: ${{ steps.selective-checks.outputs.needs-mypy }}
+ only-new-ui-files: ${{ steps.selective-checks.outputs.only-new-ui-files }}
+ postgres-exclude: ${{ steps.selective-checks.outputs.postgres-exclude }}
+ postgres-versions: ${{ steps.selective-checks.outputs.postgres-versions }}
+ prod-image-build: ${{ steps.selective-checks.outputs.prod-image-build }}
+ # yamllint disable rule:line-length
+ providers-compatibility-tests-matrix: ${{ steps.selective-checks.outputs.providers-compatibility-tests-matrix }}
+ providers-test-types-list-as-string: >-
+ ${{ steps.selective-checks.outputs.providers-test-types-list-as-string }}
pull-request-labels: ${{ steps.source-run-info.outputs.pr-labels }}
- in-workflow-build: ${{ steps.source-run-info.outputs.in-workflow-build }}
- build-job-description: ${{ steps.source-run-info.outputs.build-job-description }}
- testable-integrations: ${{ steps.selective-checks.outputs.testable-integrations }}
- canary-run: ${{ steps.source-run-info.outputs.canary-run }}
+ python-versions-list-as-string: ${{ steps.selective-checks.outputs.python-versions-list-as-string }}
+ python-versions: ${{ steps.selective-checks.outputs.python-versions }}
+ run-amazon-tests: ${{ steps.selective-checks.outputs.run-amazon-tests }}
run-coverage: ${{ steps.source-run-info.outputs.run-coverage }}
+ run-kubernetes-tests: ${{ steps.selective-checks.outputs.run-kubernetes-tests }}
+ run-task-sdk-tests: ${{ steps.selective-checks.outputs.run-task-sdk-tests }}
+ run-system-tests: ${{ steps.selective-checks.outputs.run-system-tests }}
+ run-tests: ${{ steps.selective-checks.outputs.run-tests }}
+ run-ui-tests: ${{ steps.selective-checks.outputs.run-ui-tests }}
+ run-www-tests: ${{ steps.selective-checks.outputs.run-www-tests }}
+ runs-on-as-json-default: ${{ steps.selective-checks.outputs.runs-on-as-json-default }}
+ runs-on-as-json-docs-build: ${{ steps.selective-checks.outputs.runs-on-as-json-docs-build }}
+ runs-on-as-json-public: ${{ steps.selective-checks.outputs.runs-on-as-json-public }}
+ runs-on-as-json-self-hosted-asf: ${{ steps.selective-checks.outputs.runs-on-as-json-self-hosted-asf }}
+ runs-on-as-json-self-hosted: ${{ steps.selective-checks.outputs.runs-on-as-json-self-hosted }}
+ selected-providers-list-as-string: >-
+ ${{ steps.selective-checks.outputs.selected-providers-list-as-string }}
+ skip-pre-commits: ${{ steps.selective-checks.outputs.skip-pre-commits }}
+ skip-providers-tests: ${{ steps.selective-checks.outputs.skip-providers-tests }}
+ source-head-repo: ${{ steps.source-run-info.outputs.source-head-repo }}
+ sqlite-exclude: ${{ steps.selective-checks.outputs.sqlite-exclude }}
+ test-groups: ${{ steps.selective-checks.outputs.test-groups }}
+ testable-core-integrations: ${{ steps.selective-checks.outputs.testable-core-integrations }}
+ testable-providers-integrations: ${{ steps.selective-checks.outputs.testable-providers-integrations }}
+ use-uv: ${{ steps.selective-checks.outputs.force-pip == 'true' && 'false' || 'true' }}
+ upgrade-to-newer-dependencies: ${{ steps.selective-checks.outputs.upgrade-to-newer-dependencies }}
steps:
- name: "Cleanup repo"
shell: bash
@@ -147,6 +155,9 @@ jobs:
persist-credentials: false
- name: "Install Breeze"
uses: ./.github/actions/breeze
+ with:
+ use-uv: ${{ inputs.use-uv }}
+ id: breeze
- name: "Get information about the Workflow"
id: source-run-info
run: breeze ci get-workflow-info 2>> ${GITHUB_OUTPUT}
@@ -158,7 +169,6 @@ jobs:
PR_LABELS: "${{ steps.source-run-info.outputs.pr-labels }}"
COMMIT_REF: "${{ github.sha }}"
VERBOSE: "false"
-
run: breeze ci selective-check 2>> ${GITHUB_OUTPUT}
- name: env
run: printenv
@@ -172,6 +182,7 @@ jobs:
uses: ./.github/workflows/basic-tests.yml
with:
runs-on-as-json-public: ${{ needs.build-info.outputs.runs-on-as-json-public }}
+ run-ui-tests: ${{needs.build-info.outputs.run-ui-tests}}
run-www-tests: ${{needs.build-info.outputs.run-www-tests}}
needs-api-codegen: ${{needs.build-info.outputs.needs-api-codegen}}
default-python-version: ${{needs.build-info.outputs.default-python-version}}
@@ -179,83 +190,46 @@ jobs:
skip-pre-commits: ${{needs.build-info.outputs.skip-pre-commits}}
canary-run: ${{needs.build-info.outputs.canary-run}}
latest-versions-only: ${{needs.build-info.outputs.latest-versions-only}}
- enable-aip-44: "false"
+ use-uv: ${{needs.build-info.outputs.use-uv}}
build-ci-images:
- name: >
- ${{ needs.build-info.outputs.in-workflow-build == 'true' && 'Build' || 'Skip building' }}
- CI images in-workflow
+ name: Build CI images
needs: [build-info]
uses: ./.github/workflows/ci-image-build.yml
permissions:
contents: read
# This write is only given here for `push` events from "apache/airflow" repo. It is not given for PRs
# from forks. This is to prevent malicious PRs from creating images in the "apache/airflow" repo.
- # For regular build for PRS this "build-prod-images" workflow will be skipped anyway by the
- # "in-workflow-build" condition
packages: write
secrets: inherit
with:
runs-on-as-json-public: ${{ needs.build-info.outputs.runs-on-as-json-public }}
runs-on-as-json-self-hosted: ${{ needs.build-info.outputs.runs-on-as-json-self-hosted }}
- do-build: ${{ needs.build-info.outputs.in-workflow-build }}
- image-tag: ${{ needs.build-info.outputs.image-tag }}
platform: "linux/amd64"
+ push-image: "false"
+ upload-image-artifact: "true"
+ upload-mount-cache-artifact: ${{ needs.build-info.outputs.canary-run }}
python-versions: ${{ needs.build-info.outputs.python-versions }}
branch: ${{ needs.build-info.outputs.default-branch }}
- use-uv: ${{ needs.build-info.outputs.force-pip && 'false' || 'true' }}
+ use-uv: ${{ needs.build-info.outputs.use-uv }}
upgrade-to-newer-dependencies: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
constraints-branch: ${{ needs.build-info.outputs.default-constraints-branch }}
docker-cache: ${{ needs.build-info.outputs.docker-cache }}
-
- wait-for-ci-images:
- timeout-minutes: 120
- name: "Wait for CI images"
- runs-on: ${{ fromJSON(needs.build-info.outputs.runs-on-as-json-public) }}
- needs: [build-info, build-ci-images]
+ disable-airflow-repo-cache: ${{ needs.build-info.outputs.disable-airflow-repo-cache }}
if: needs.build-info.outputs.ci-image-build == 'true'
- env:
- BACKEND: sqlite
- # Force more parallelism for pull even on public images
- PARALLELISM: 6
- INCLUDE_SUCCESS_OUTPUTS: "${{needs.build-info.outputs.include-success-outputs}}"
- steps:
- - name: "Cleanup repo"
- shell: bash
- run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
- if: needs.build-info.outputs.in-workflow-build == 'false'
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
- uses: actions/checkout@v4
- with:
- persist-credentials: false
- if: needs.build-info.outputs.in-workflow-build == 'false'
- - name: "Cleanup docker"
- run: ./scripts/ci/cleanup_docker.sh
- if: needs.build-info.outputs.in-workflow-build == 'false'
- - name: "Install Breeze"
- uses: ./.github/actions/breeze
- if: needs.build-info.outputs.in-workflow-build == 'false'
- - name: Login to ghcr.io
- run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
- if: needs.build-info.outputs.in-workflow-build == 'false'
- - name: Wait for CI images ${{ env.PYTHON_VERSIONS }}:${{ needs.build-info.outputs.image-tag }}
- id: wait-for-images
- run: breeze ci-image pull --run-in-parallel --wait-for-image --tag-as-latest
- env:
- PYTHON_VERSIONS: ${{ needs.build-info.outputs.python-versions-list-as-string }}
- DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
- if: needs.build-info.outputs.in-workflow-build == 'false'
additional-ci-image-checks:
name: "Additional CI image checks"
- needs: [build-info, wait-for-ci-images]
+ needs: [build-info, build-ci-images]
uses: ./.github/workflows/additional-ci-image-checks.yml
+ permissions:
+ contents: read
+ packages: write
if: needs.build-info.outputs.canary-run == 'true'
with:
runs-on-as-json-default: ${{ needs.build-info.outputs.runs-on-as-json-default }}
runs-on-as-json-public: ${{ needs.build-info.outputs.runs-on-as-json-public }}
runs-on-as-json-self-hosted: ${{ needs.build-info.outputs.runs-on-as-json-self-hosted }}
- image-tag: ${{ needs.build-info.outputs.image-tag }}
python-versions: ${{ needs.build-info.outputs.python-versions }}
branch: ${{ needs.build-info.outputs.default-branch }}
constraints-branch: ${{ needs.build-info.outputs.default-constraints-branch }}
@@ -263,16 +237,16 @@ jobs:
upgrade-to-newer-dependencies: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
skip-pre-commits: ${{ needs.build-info.outputs.skip-pre-commits }}
docker-cache: ${{ needs.build-info.outputs.docker-cache }}
+ disable-airflow-repo-cache: ${{ needs.build-info.outputs.disable-airflow-repo-cache }}
canary-run: ${{ needs.build-info.outputs.canary-run }}
latest-versions-only: ${{ needs.build-info.outputs.latest-versions-only }}
include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }}
debug-resources: ${{ needs.build-info.outputs.debug-resources }}
- use-uv: ${{ needs.build-info.outputs.force-pip && 'false' || 'true' }}
-
+ use-uv: ${{ needs.build-info.outputs.use-uv }}
generate-constraints:
name: "Generate constraints"
- needs: [build-info, wait-for-ci-images]
+ needs: [build-info, build-ci-images]
uses: ./.github/workflows/generate-constraints.yml
if: needs.build-info.outputs.ci-image-build == 'true'
with:
@@ -281,19 +255,18 @@ jobs:
# generate no providers constraints only in canary builds - they take quite some time to generate
# they are not needed for regular builds, they are only needed to update constraints in canaries
generate-no-providers-constraints: ${{ needs.build-info.outputs.canary-run }}
- image-tag: ${{ needs.build-info.outputs.image-tag }}
chicken-egg-providers: ${{ needs.build-info.outputs.chicken-egg-providers }}
debug-resources: ${{ needs.build-info.outputs.debug-resources }}
+ use-uv: ${{ needs.build-info.outputs.use-uv }}
- static-checks-mypy-docs:
- name: "Static checks, mypy, docs"
- needs: [build-info, wait-for-ci-images]
- uses: ./.github/workflows/static-checks-mypy-docs.yml
+ ci-image-checks:
+ name: "CI image checks"
+ needs: [build-info, build-ci-images]
+ uses: ./.github/workflows/ci-image-checks.yml
secrets: inherit
with:
runs-on-as-json-default: ${{ needs.build-info.outputs.runs-on-as-json-default }}
runs-on-as-json-docs-build: ${{ needs.build-info.outputs.runs-on-as-json-docs-build }}
- image-tag: ${{ needs.build-info.outputs.image-tag }}
needs-mypy: ${{ needs.build-info.outputs.needs-mypy }}
mypy-checks: ${{ needs.build-info.outputs.mypy-checks }}
python-versions-list-as-string: ${{ needs.build-info.outputs.python-versions-list-as-string }}
@@ -310,34 +283,39 @@ jobs:
include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }}
debug-resources: ${{ needs.build-info.outputs.debug-resources }}
docs-build: ${{ needs.build-info.outputs.docs-build }}
+ needs-api-codegen: ${{ needs.build-info.outputs.needs-api-codegen }}
+ default-postgres-version: ${{ needs.build-info.outputs.default-postgres-version }}
+ run-coverage: ${{ needs.build-info.outputs.run-coverage }}
+ use-uv: ${{ needs.build-info.outputs.use-uv }}
providers:
- name: "Provider checks"
- uses: ./.github/workflows/check-providers.yml
- needs: [build-info, wait-for-ci-images]
+ name: "Provider packages tests"
+ uses: ./.github/workflows/test-provider-packages.yml
+ needs: [build-info, build-ci-images]
permissions:
contents: read
packages: read
secrets: inherit
if: >
- needs.build-info.outputs.skip-provider-tests != 'true' &&
+ needs.build-info.outputs.skip-providers-tests != 'true' &&
needs.build-info.outputs.latest-versions-only != 'true'
with:
runs-on-as-json-default: ${{ needs.build-info.outputs.runs-on-as-json-default }}
- image-tag: ${{ needs.build-info.outputs.image-tag }}
canary-run: ${{ needs.build-info.outputs.canary-run }}
default-python-version: ${{ needs.build-info.outputs.default-python-version }}
upgrade-to-newer-dependencies: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
- affected-providers-list-as-string: ${{ needs.build-info.outputs.affected-providers-list-as-string }}
- providers-compatibility-checks: ${{ needs.build-info.outputs.providers-compatibility-checks }}
- skip-provider-tests: ${{ needs.build-info.outputs.skip-provider-tests }}
+ selected-providers-list-as-string: ${{ needs.build-info.outputs.selected-providers-list-as-string }}
+ # yamllint disable rule:line-length
+ providers-compatibility-tests-matrix: ${{ needs.build-info.outputs.providers-compatibility-tests-matrix }}
+ skip-providers-tests: ${{ needs.build-info.outputs.skip-providers-tests }}
python-versions: ${{ needs.build-info.outputs.python-versions }}
providers-test-types-list-as-string: ${{ needs.build-info.outputs.providers-test-types-list-as-string }}
+ use-uv: ${{ needs.build-info.outputs.use-uv }}
tests-helm:
name: "Helm tests"
uses: ./.github/workflows/helm-tests.yml
- needs: [build-info, wait-for-ci-images]
+ needs: [build-info, build-ci-images]
permissions:
contents: read
packages: read
@@ -346,8 +324,8 @@ jobs:
runs-on-as-json-default: ${{ needs.build-info.outputs.runs-on-as-json-default }}
runs-on-as-json-public: ${{ needs.build-info.outputs.runs-on-as-json-public }}
helm-test-packages: ${{ needs.build-info.outputs.helm-test-packages }}
- image-tag: ${{ needs.build-info.outputs.image-tag }}
default-python-version: ${{ needs.build-info.outputs.default-python-version }}
+ use-uv: ${{ needs.build-info.outputs.use-uv }}
if: >
needs.build-info.outputs.needs-helm-tests == 'true' &&
needs.build-info.outputs.default-branch == 'main' &&
@@ -356,7 +334,7 @@ jobs:
tests-postgres:
name: "Postgres tests"
uses: ./.github/workflows/run-unit-tests.yml
- needs: [build-info, wait-for-ci-images]
+ needs: [build-info, build-ci-images]
permissions:
contents: read
packages: read
@@ -366,21 +344,24 @@ jobs:
backend: "postgres"
test-name: "Postgres"
test-scope: "DB"
- image-tag: ${{ needs.build-info.outputs.image-tag }}
+ test-groups: ${{ needs.build-info.outputs.test-groups }}
python-versions: ${{ needs.build-info.outputs.python-versions }}
backend-versions: ${{ needs.build-info.outputs.postgres-versions }}
+ excluded-providers-as-string: ${{ needs.build-info.outputs.excluded-providers-as-string }}
excludes: ${{ needs.build-info.outputs.postgres-exclude }}
- parallel-test-types-list-as-string: ${{ needs.build-info.outputs.parallel-test-types-list-as-string }}
+ core-test-types-list-as-string: ${{ needs.build-info.outputs.core-test-types-list-as-string }}
+ providers-test-types-list-as-string: ${{ needs.build-info.outputs.providers-test-types-list-as-string }}
include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }}
run-migration-tests: "true"
run-coverage: ${{ needs.build-info.outputs.run-coverage }}
debug-resources: ${{ needs.build-info.outputs.debug-resources }}
+ use-uv: ${{ needs.build-info.outputs.use-uv }}
if: needs.build-info.outputs.run-tests == 'true'
tests-mysql:
name: "MySQL tests"
uses: ./.github/workflows/run-unit-tests.yml
- needs: [build-info, wait-for-ci-images]
+ needs: [build-info, build-ci-images]
permissions:
contents: read
packages: read
@@ -390,21 +371,24 @@ jobs:
backend: "mysql"
test-name: "MySQL"
test-scope: "DB"
- image-tag: ${{ needs.build-info.outputs.image-tag }}
+ test-groups: ${{ needs.build-info.outputs.test-groups }}
python-versions: ${{ needs.build-info.outputs.python-versions }}
backend-versions: ${{ needs.build-info.outputs.mysql-versions }}
+ excluded-providers-as-string: ${{ needs.build-info.outputs.excluded-providers-as-string }}
excludes: ${{ needs.build-info.outputs.mysql-exclude }}
- parallel-test-types-list-as-string: ${{ needs.build-info.outputs.parallel-test-types-list-as-string }}
+ core-test-types-list-as-string: ${{ needs.build-info.outputs.core-test-types-list-as-string }}
+ providers-test-types-list-as-string: ${{ needs.build-info.outputs.providers-test-types-list-as-string }}
include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }}
run-coverage: ${{ needs.build-info.outputs.run-coverage }}
run-migration-tests: "true"
debug-resources: ${{ needs.build-info.outputs.debug-resources }}
+ use-uv: ${{ needs.build-info.outputs.use-uv }}
if: needs.build-info.outputs.run-tests == 'true'
tests-sqlite:
name: "Sqlite tests"
uses: ./.github/workflows/run-unit-tests.yml
- needs: [build-info, wait-for-ci-images]
+ needs: [build-info, build-ci-images]
permissions:
contents: read
packages: read
@@ -415,22 +399,25 @@ jobs:
test-name: "Sqlite"
test-name-separator: ""
test-scope: "DB"
- image-tag: ${{ needs.build-info.outputs.image-tag }}
+ test-groups: ${{ needs.build-info.outputs.test-groups }}
python-versions: ${{ needs.build-info.outputs.python-versions }}
# No versions for sqlite
backend-versions: "['']"
+ excluded-providers-as-string: ${{ needs.build-info.outputs.excluded-providers-as-string }}
excludes: ${{ needs.build-info.outputs.sqlite-exclude }}
- parallel-test-types-list-as-string: ${{ needs.build-info.outputs.parallel-test-types-list-as-string }}
+ core-test-types-list-as-string: ${{ needs.build-info.outputs.core-test-types-list-as-string }}
+ providers-test-types-list-as-string: ${{ needs.build-info.outputs.providers-test-types-list-as-string }}
include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }}
run-coverage: ${{ needs.build-info.outputs.run-coverage }}
run-migration-tests: "true"
debug-resources: ${{ needs.build-info.outputs.debug-resources }}
+ use-uv: ${{ needs.build-info.outputs.use-uv }}
if: needs.build-info.outputs.run-tests == 'true'
tests-non-db:
name: "Non-DB tests"
uses: ./.github/workflows/run-unit-tests.yml
- needs: [build-info, wait-for-ci-images]
+ needs: [build-info, build-ci-images]
permissions:
contents: read
packages: read
@@ -441,21 +428,24 @@ jobs:
test-name: ""
test-name-separator: ""
test-scope: "Non-DB"
- image-tag: ${{ needs.build-info.outputs.image-tag }}
+ test-groups: ${{ needs.build-info.outputs.test-groups }}
python-versions: ${{ needs.build-info.outputs.python-versions }}
# No versions for non-db
backend-versions: "['']"
+ excluded-providers-as-string: ${{ needs.build-info.outputs.excluded-providers-as-string }}
excludes: ${{ needs.build-info.outputs.sqlite-exclude }}
- parallel-test-types-list-as-string: ${{ needs.build-info.outputs.parallel-test-types-list-as-string }}
+ core-test-types-list-as-string: ${{ needs.build-info.outputs.core-test-types-list-as-string }}
+ providers-test-types-list-as-string: ${{ needs.build-info.outputs.providers-test-types-list-as-string }}
include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }}
run-coverage: ${{ needs.build-info.outputs.run-coverage }}
debug-resources: ${{ needs.build-info.outputs.debug-resources }}
+ use-uv: ${{ needs.build-info.outputs.use-uv }}
if: needs.build-info.outputs.run-tests == 'true'
tests-special:
name: "Special tests"
uses: ./.github/workflows/special-tests.yml
- needs: [build-info, wait-for-ci-images]
+ needs: [build-info, build-ci-images]
permissions:
contents: read
packages: read
@@ -466,41 +456,47 @@ jobs:
needs.build-info.outputs.upgrade-to-newer-dependencies != 'false' ||
needs.build-info.outputs.full-tests-needed == 'true')
with:
+ test-groups: ${{ needs.build-info.outputs.test-groups }}
default-branch: ${{ needs.build-info.outputs.default-branch }}
runs-on-as-json-default: ${{ needs.build-info.outputs.runs-on-as-json-default }}
- image-tag: ${{ needs.build-info.outputs.image-tag }}
- parallel-test-types-list-as-string: ${{ needs.build-info.outputs.parallel-test-types-list-as-string }}
+ core-test-types-list-as-string: ${{ needs.build-info.outputs.core-test-types-list-as-string }}
+ providers-test-types-list-as-string: ${{ needs.build-info.outputs.providers-test-types-list-as-string }}
run-coverage: ${{ needs.build-info.outputs.run-coverage }}
default-python-version: ${{ needs.build-info.outputs.default-python-version }}
python-versions: ${{ needs.build-info.outputs.python-versions }}
default-postgres-version: ${{ needs.build-info.outputs.default-postgres-version }}
+ excluded-providers-as-string: ${{ needs.build-info.outputs.excluded-providers-as-string }}
canary-run: ${{ needs.build-info.outputs.canary-run }}
upgrade-to-newer-dependencies: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
+ include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }}
debug-resources: ${{ needs.build-info.outputs.debug-resources }}
+ use-uv: ${{ needs.build-info.outputs.use-uv }}
- tests-integration:
- name: Integration Tests
- needs: [build-info, wait-for-ci-images]
- uses: ./.github/workflows/integration-tests.yml
+ tests-integration-system:
+ name: Integration and System Tests
+ needs: [build-info, build-ci-images]
+ uses: ./.github/workflows/integration-system-tests.yml
permissions:
contents: read
packages: read
secrets: inherit
with:
runs-on-as-json-public: ${{ needs.build-info.outputs.runs-on-as-json-public }}
- image-tag: ${{ needs.build-info.outputs.image-tag }}
- testable-integrations: ${{ needs.build-info.outputs.testable-integrations }}
+ testable-core-integrations: ${{ needs.build-info.outputs.testable-core-integrations }}
+ testable-providers-integrations: ${{ needs.build-info.outputs.testable-providers-integrations }}
+ run-system-tests: ${{ needs.build-info.outputs.run-tests }}
default-python-version: ${{ needs.build-info.outputs.default-python-version }}
default-postgres-version: ${{ needs.build-info.outputs.default-postgres-version }}
default-mysql-version: ${{ needs.build-info.outputs.default-mysql-version }}
- skip-provider-tests: ${{ needs.build-info.outputs.skip-provider-tests }}
+ skip-providers-tests: ${{ needs.build-info.outputs.skip-providers-tests }}
run-coverage: ${{ needs.build-info.outputs.run-coverage }}
debug-resources: ${{ needs.build-info.outputs.debug-resources }}
+ use-uv: ${{ needs.build-info.outputs.use-uv }}
if: needs.build-info.outputs.run-tests == 'true'
tests-with-lowest-direct-resolution:
- name: "Lowest direct dependency resolution tests"
- needs: [build-info, wait-for-ci-images]
+ name: "Lowest direct dependency providers tests"
+ needs: [build-info, build-ci-images]
uses: ./.github/workflows/run-unit-tests.yml
permissions:
contents: read
@@ -513,129 +509,107 @@ jobs:
test-name: "LowestDeps-Postgres"
force-lowest-dependencies: "true"
test-scope: "All"
+ test-groups: ${{ needs.build-info.outputs.test-groups }}
backend: "postgres"
- image-tag: ${{ needs.build-info.outputs.image-tag }}
python-versions: ${{ needs.build-info.outputs.python-versions }}
backend-versions: "['${{ needs.build-info.outputs.default-postgres-version }}']"
+ excluded-providers-as-string: ${{ needs.build-info.outputs.excluded-providers-as-string }}
excludes: "[]"
- parallel-test-types-list-as-string: ${{ needs.build-info.outputs.separate-test-types-list-as-string }}
+ core-test-types-list-as-string: ${{ needs.build-info.outputs.core-test-types-list-as-string }}
+ # yamllint disable rule:line-length
+ providers-test-types-list-as-string: ${{ needs.build-info.outputs.individual-providers-test-types-list-as-string }}
include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }}
run-coverage: ${{ needs.build-info.outputs.run-coverage }}
debug-resources: ${{ needs.build-info.outputs.debug-resources }}
monitor-delay-time-in-seconds: 120
+ use-uv: ${{ needs.build-info.outputs.use-uv }}
build-prod-images:
- name: >
- ${{ needs.build-info.outputs.in-workflow-build == 'true' && 'Build' || 'Skip building' }}
- PROD images in-workflow
+ name: Build PROD images
needs: [build-info, build-ci-images, generate-constraints]
uses: ./.github/workflows/prod-image-build.yml
permissions:
contents: read
# This write is only given here for `push` events from "apache/airflow" repo. It is not given for PRs
# from forks. This is to prevent malicious PRs from creating images in the "apache/airflow" repo.
- # For regular build for PRS this "build-prod-images" workflow will be skipped anyway by the
- # "in-workflow-build" condition
packages: write
secrets: inherit
with:
runs-on-as-json-public: ${{ needs.build-info.outputs.runs-on-as-json-public }}
build-type: "Regular"
- do-build: ${{ needs.build-info.outputs.in-workflow-build }}
- upload-package-artifact: "true"
- image-tag: ${{ needs.build-info.outputs.image-tag }}
platform: "linux/amd64"
+ push-image: "false"
+ upload-image-artifact: "true"
+ upload-package-artifact: "true"
python-versions: ${{ needs.build-info.outputs.python-versions }}
default-python-version: ${{ needs.build-info.outputs.default-python-version }}
branch: ${{ needs.build-info.outputs.default-branch }}
- push-image: "true"
- use-uv: ${{ needs.build-info.outputs.force-pip && 'false' || 'true' }}
+ use-uv: ${{ needs.build-info.outputs.use-uv }}
build-provider-packages: ${{ needs.build-info.outputs.default-branch == 'main' }}
upgrade-to-newer-dependencies: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
chicken-egg-providers: ${{ needs.build-info.outputs.chicken-egg-providers }}
constraints-branch: ${{ needs.build-info.outputs.default-constraints-branch }}
docker-cache: ${{ needs.build-info.outputs.docker-cache }}
-
- wait-for-prod-images:
- timeout-minutes: 80
- name: "Wait for PROD images"
- runs-on: ${{ fromJSON(needs.build-info.outputs.runs-on-as-json-public) }}
- needs: [build-info, wait-for-ci-images, build-prod-images]
- if: needs.build-info.outputs.prod-image-build == 'true'
- env:
- BACKEND: sqlite
- PYTHON_MAJOR_MINOR_VERSION: "${{needs.build-info.outputs.default-python-version}}"
- # Force more parallelism for pull on public images
- PARALLELISM: 6
- INCLUDE_SUCCESS_OUTPUTS: "${{needs.build-info.outputs.include-success-outputs}}"
- IMAGE_TAG: ${{ needs.build-info.outputs.image-tag }}
- steps:
- - name: "Cleanup repo"
- shell: bash
- run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
- if: needs.build-info.outputs.in-workflow-build == 'false'
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
- uses: actions/checkout@v4
- with:
- persist-credentials: false
- if: needs.build-info.outputs.in-workflow-build == 'false'
- - name: "Cleanup docker"
- run: ./scripts/ci/cleanup_docker.sh
- if: needs.build-info.outputs.in-workflow-build == 'false'
- - name: "Install Breeze"
- uses: ./.github/actions/breeze
- if: needs.build-info.outputs.in-workflow-build == 'false'
- - name: Login to ghcr.io
- run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
- if: needs.build-info.outputs.in-workflow-build == 'false'
- - name: Wait for PROD images ${{ env.PYTHON_VERSIONS }}:${{ needs.build-info.outputs.image-tag }}
- # We wait for the images to be available either from "build-images.yml' run as pull_request_target
- # or from build-prod-images (or build-prod-images-release-branch) above.
- # We are utilising single job to wait for all images because this job merely waits
- # For the images to be available.
- run: breeze prod-image pull --wait-for-image --run-in-parallel
- env:
- PYTHON_VERSIONS: ${{ needs.build-info.outputs.python-versions-list-as-string }}
- DEBUG_RESOURCES: ${{ needs.build-info.outputs.debug-resources }}
- if: needs.build-info.outputs.in-workflow-build == 'false'
+ disable-airflow-repo-cache: ${{ needs.build-info.outputs.disable-airflow-repo-cache }}
+ prod-image-build: ${{ needs.build-info.outputs.prod-image-build }}
additional-prod-image-tests:
name: "Additional PROD image tests"
- needs: [build-info, wait-for-prod-images, generate-constraints]
+ needs: [build-info, build-prod-images, generate-constraints]
uses: ./.github/workflows/additional-prod-image-tests.yml
with:
runs-on-as-json-public: ${{ needs.build-info.outputs.runs-on-as-json-public }}
default-branch: ${{ needs.build-info.outputs.default-branch }}
constraints-branch: ${{ needs.build-info.outputs.default-constraints-branch }}
- image-tag: ${{ needs.build-info.outputs.image-tag }}
upgrade-to-newer-dependencies: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
chicken-egg-providers: ${{ needs.build-info.outputs.chicken-egg-providers }}
docker-cache: ${{ needs.build-info.outputs.docker-cache }}
+ disable-airflow-repo-cache: ${{ needs.build-info.outputs.disable-airflow-repo-cache }}
default-python-version: ${{ needs.build-info.outputs.default-python-version }}
canary-run: ${{ needs.build-info.outputs.canary-run }}
+ use-uv: ${{ needs.build-info.outputs.use-uv }}
if: needs.build-info.outputs.prod-image-build == 'true'
tests-kubernetes:
name: "Kubernetes tests"
uses: ./.github/workflows/k8s-tests.yml
- needs: [build-info, wait-for-prod-images]
+ needs: [build-info, build-prod-images]
permissions:
contents: read
packages: read
secrets: inherit
with:
+ platform: "linux/amd64"
runs-on-as-json-default: ${{ needs.build-info.outputs.runs-on-as-json-default }}
- image-tag: ${{ needs.build-info.outputs.image-tag }}
python-versions-list-as-string: ${{ needs.build-info.outputs.python-versions-list-as-string }}
- kubernetes-versions-list-as-string: ${{ needs.build-info.outputs.kubernetes-versions-list-as-string }}
- kubernetes-combos-list-as-string: ${{ needs.build-info.outputs.kubernetes-combos-list-as-string }}
include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }}
- use-uv: ${{ needs.build-info.outputs.force-pip && 'false' || 'true' }}
+ use-uv: ${{ needs.build-info.outputs.use-uv }}
debug-resources: ${{ needs.build-info.outputs.debug-resources }}
+ kubernetes-combos: ${{ needs.build-info.outputs.kubernetes-combos }}
if: >
( needs.build-info.outputs.run-kubernetes-tests == 'true' ||
needs.build-info.outputs.needs-helm-tests == 'true')
+ tests-task-sdk:
+ name: "Task SDK tests"
+ uses: ./.github/workflows/task-sdk-tests.yml
+ needs: [build-info, build-ci-images]
+ permissions:
+ contents: read
+ packages: read
+ secrets: inherit
+ with:
+ runs-on-as-json-default: ${{ needs.build-info.outputs.runs-on-as-json-default }}
+ default-python-version: ${{ needs.build-info.outputs.default-python-version }}
+ python-versions: ${{ needs.build-info.outputs.python-versions }}
+ run-task-sdk-tests: ${{ needs.build-info.outputs.run-task-sdk-tests }}
+ use-uv: ${{ needs.build-info.outputs.use-uv }}
+ canary-run: ${{ needs.build-info.outputs.canary-run }}
+ if: >
+ ( needs.build-info.outputs.run-task-sdk-tests == 'true' ||
+ needs.build-info.outputs.run-tests == 'true' &&
+ needs.build-info.outputs.only-new-ui-files != 'true')
+
finalize-tests:
name: Finalize tests
permissions:
@@ -645,26 +619,59 @@ jobs:
needs:
- build-info
- generate-constraints
- - wait-for-ci-images
- - wait-for-prod-images
- - static-checks-mypy-docs
+ - ci-image-checks
- tests-sqlite
- tests-mysql
- tests-postgres
- tests-non-db
- - tests-integration
+ - tests-integration-system
+ - build-prod-images
uses: ./.github/workflows/finalize-tests.yml
with:
runs-on-as-json-public: ${{ needs.build-info.outputs.runs-on-as-json-public }}
runs-on-as-json-self-hosted: ${{ needs.build-info.outputs.runs-on-as-json-self-hosted }}
- image-tag: ${{ needs.build-info.outputs.image-tag }}
python-versions: ${{ needs.build-info.outputs.python-versions }}
python-versions-list-as-string: ${{ needs.build-info.outputs.python-versions-list-as-string }}
branch: ${{ needs.build-info.outputs.default-branch }}
constraints-branch: ${{ needs.build-info.outputs.default-constraints-branch }}
default-python-version: ${{ needs.build-info.outputs.default-python-version }}
- in-workflow-build: ${{ needs.build-info.outputs.in-workflow-build }}
upgrade-to-newer-dependencies: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }}
docker-cache: ${{ needs.build-info.outputs.docker-cache }}
+ disable-airflow-repo-cache: ${{ needs.build-info.outputs.disable-airflow-repo-cache }}
canary-run: ${{ needs.build-info.outputs.canary-run }}
+ use-uv: ${{ needs.build-info.outputs.use-uv }}
+ debug-resources: ${{ needs.build-info.outputs.debug-resources }}
+
+ notify-slack-failure:
+ name: "Notify Slack on Failure"
+ needs:
+ - basic-tests
+ - additional-ci-image-checks
+ - providers
+ - tests-helm
+ - tests-special
+ - tests-with-lowest-direct-resolution
+ - additional-prod-image-tests
+ - tests-kubernetes
+ - tests-task-sdk
+ - finalize-tests
+ if: github.event_name == 'schedule' && failure() && github.run_attempt == 1
+ runs-on: ["ubuntu-22.04"]
+ steps:
+ - name: Notify Slack
+ id: slack
+ uses: slackapi/slack-github-action@485a9d42d3a73031f12ec201c457e2162c45d02d # v2.0.0
+ with:
+ method: chat.postMessage
+ token: ${{ env.SLACK_BOT_TOKEN }}
+ # yamllint disable rule:line-length
+ payload: |
+ channel: "internal-airflow-ci-cd"
+ text: "🚨🕒 Scheduled CI Failure Alert 🕒🚨\n\n*Details:* <https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}|View the failure log>"
+ blocks:
+ - type: "section"
+ text:
+ type: "mrkdwn"
+ text: "🚨🕒 Scheduled CI Failure Alert 🕒🚨\n\n*Details:* <https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}|View the failure log>"
+ # yamllint enable rule:line-length
diff --git a/.github/workflows/codeql-analysis.yml b/.github/workflows/codeql-analysis.yml
index ec608192a70..3f4d7221fc6 100644
--- a/.github/workflows/codeql-analysis.yml
+++ b/.github/workflows/codeql-analysis.yml
@@ -19,6 +19,8 @@
name: "CodeQL"
on: # yamllint disable-line rule:truthy
+ pull_request:
+ branches: ['main', 'v[0-9]+-[0-9]+-test', 'v[0-9]+-[0-9]+-stable']
push:
branches: [main]
schedule:
@@ -34,6 +36,8 @@ jobs:
selective-checks:
name: Selective checks
runs-on: ["ubuntu-22.04"]
+ env:
+ GITHUB_CONTEXT: ${{ toJson(github) }}
outputs:
needs-python-scans: ${{ steps.selective-checks.outputs.needs-python-scans }}
needs-javascript-scans: ${{ steps.selective-checks.outputs.needs-javascript-scans }}
@@ -45,9 +49,17 @@ jobs:
persist-credentials: false
- name: "Install Breeze"
uses: ./.github/actions/breeze
+ with:
+ use-uv: "true"
+ - name: "Get information about the Workflow"
+ id: source-run-info
+ run: breeze ci get-workflow-info 2>> ${GITHUB_OUTPUT}
+ env:
+ SKIP_BREEZE_SELF_UPGRADE_CHECK: "true"
- name: Selective checks
id: selective-checks
env:
+ PR_LABELS: "${{ steps.source-run-info.outputs.pr-labels }}"
COMMIT_REF: "${{ github.sha }}"
VERBOSE: "false"
run: breeze ci selective-check 2>> ${GITHUB_OUTPUT}
@@ -78,7 +90,7 @@ jobs:
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
- uses: github/codeql-action/init@v2
+ uses: github/codeql-action/init@v3
with:
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
@@ -92,13 +104,13 @@ jobs:
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild
- uses: github/codeql-action/autobuild@v2
+ uses: github/codeql-action/autobuild@v3
if: |
matrix.language == 'python' && needs.selective-checks.outputs.needs-python-scans == 'true' ||
matrix.language == 'javascript' && needs.selective-checks.outputs.needs-javascript-scans == 'true'
- name: Perform CodeQL Analysis
- uses: github/codeql-action/analyze@v2
+ uses: github/codeql-action/analyze@v3
if: |
matrix.language == 'python' && needs.selective-checks.outputs.needs-python-scans == 'true' ||
matrix.language == 'javascript' && needs.selective-checks.outputs.needs-javascript-scans == 'true'
diff --git a/.github/workflows/finalize-tests.yml b/.github/workflows/finalize-tests.yml
index a460dbe151a..1d0ac8a600c 100644
--- a/.github/workflows/finalize-tests.yml
+++ b/.github/workflows/finalize-tests.yml
@@ -28,10 +28,6 @@ on: # yamllint disable-line rule:truthy
description: "The array of labels (in json form) determining self-hosted runners."
required: true
type: string
- image-tag:
- description: "Tag to set for the image"
- required: true
- type: string
python-versions:
description: "JSON-formatted array of Python versions to test"
required: true
@@ -52,10 +48,6 @@ on: # yamllint disable-line rule:truthy
description: "Which version of python should be used by default"
required: true
type: string
- in-workflow-build:
- description: "Whether the build is executed as part of the workflow (true/false)"
- required: true
- type: string
upgrade-to-newer-dependencies:
description: "Whether to upgrade to newer dependencies (true/false)"
required: true
@@ -64,6 +56,10 @@ on: # yamllint disable-line rule:truthy
description: "Docker cache specification to build the image (registry, local, disabled)."
required: true
type: string
+ disable-airflow-repo-cache:
+ description: "Disable airflow repo cache read from main."
+ required: true
+ type: string
include-success-outputs:
description: "Whether to include success outputs (true/false)"
required: true
@@ -72,6 +68,14 @@ on: # yamllint disable-line rule:truthy
description: "Whether this is a canary run (true/false)"
required: true
type: string
+ use-uv:
+ description: "Whether to use uv to build the image (true/false)"
+ required: true
+ type: string
+ debug-resources:
+ description: "Whether to debug resources or not (true/false)"
+ required: true
+ type: string
jobs:
update-constraints:
runs-on: ${{ fromJSON(inputs.runs-on-as-json-public) }}
@@ -83,7 +87,6 @@ jobs:
env:
DEBUG_RESOURCES: ${{ inputs.debug-resources}}
PYTHON_VERSIONS: ${{ inputs.python-versions-list-as-string }}
- IMAGE_TAG: ${{ inputs.image-tag }}
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_USERNAME: ${{ github.actor }}
@@ -145,9 +148,10 @@ jobs:
python-versions: ${{ inputs.python-versions }}
branch: ${{ inputs.branch }}
constraints-branch: ${{ inputs.constraints-branch }}
- use-uv: ${{ needs.build-info.outputs.force-pip && 'false' || 'true' }}
+ use-uv: ${{ inputs.use-uv }}
include-success-outputs: ${{ inputs.include-success-outputs }}
docker-cache: ${{ inputs.docker-cache }}
+ disable-airflow-repo-cache: ${{ inputs.disable-airflow-repo-cache }}
if: inputs.canary-run == 'true'
# push-buildx-cache-to-github-registry-arm:
@@ -187,10 +191,14 @@ jobs:
persist-credentials: false
- name: "Cleanup docker"
run: ./scripts/ci/cleanup_docker.sh
- - name: "Download all artifacts from the current build"
+ - name: "Free up disk space"
+ shell: bash
+ run: ./scripts/tools/free_up_disk_space.sh
+ - name: "Download all test warning artifacts from the current build"
uses: actions/download-artifact@v4
with:
path: ./artifacts
+ pattern: test-warnings-*
- name: "Setup python"
uses: actions/setup-python@v5
with:
diff --git a/.github/workflows/generate-constraints.yml b/.github/workflows/generate-constraints.yml
index d6e536dfd09..740310e1cc0 100644
--- a/.github/workflows/generate-constraints.yml
+++ b/.github/workflows/generate-constraints.yml
@@ -32,10 +32,6 @@ on: # yamllint disable-line rule:truthy
description: "Whether to generate constraints without providers (true/false)"
required: true
type: string
- image-tag:
- description: "Tag to set for the image"
- required: true
- type: string
chicken-egg-providers:
description: "Space-separated list of providers that should be installed from context files"
required: true
@@ -44,6 +40,10 @@ on: # yamllint disable-line rule:truthy
description: "Whether to run in debug mode (true/false)"
required: true
type: string
+ use-uv:
+ description: "Whether to use uvloop (true/false)"
+ required: true
+ type: string
jobs:
generate-constraints:
permissions:
@@ -57,7 +57,6 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_USERNAME: ${{ github.actor }}
INCLUDE_SUCCESS_OUTPUTS: "true"
- IMAGE_TAG: ${{ inputs.image-tag }}
PYTHON_VERSIONS: ${{ inputs.python-versions-list-as-string }}
VERBOSE: "true"
VERSION_SUFFIX_FOR_PYPI: "dev0"
@@ -69,21 +68,17 @@ jobs:
uses: actions/checkout@v4
with:
persist-credentials: false
- - name: "Cleanup docker"
- run: ./scripts/ci/cleanup_docker.sh
- name: "Install Breeze"
uses: ./.github/actions/breeze
- - name: Login to ghcr.io
- run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
- - name: "\
- Pull CI images \
- ${{ inputs.python-versions-list-as-string }}:\
- ${{ inputs.image-tag }}"
- run: breeze ci-image pull --run-in-parallel --tag-as-latest
- - name: "
- Verify CI images \
- ${{ inputs.python-versions-list-as-string }}:\
- ${{ inputs.image-tag }}"
+ with:
+ use-uv: ${{ inputs.use-uv }}
+ id: breeze
+ - name: "Prepare all CI images: ${{ inputs.python-versions-list-as-string}}"
+ uses: ./.github/actions/prepare_all_ci_images
+ with:
+ platform: "linux/amd64"
+ python-versions-list-as-string: ${{ inputs.python-versions-list-as-string }}
+ - name: "Verify all CI images ${{ inputs.python-versions-list-as-string }}"
run: breeze ci-image verify --run-in-parallel
- name: "Source constraints"
shell: bash
@@ -104,22 +99,28 @@ jobs:
# from the source code, not from the PyPI because they have apache-airflow>=X.Y.Z dependency
# And when we prepare them from sources they will have apache-airflow>=X.Y.Z.dev0
shell: bash
+ env:
+ CHICKEN_EGG_PROVIDERS: ${{ inputs.chicken-egg-providers }}
run: >
breeze release-management prepare-provider-packages --include-not-ready-providers
--package-format wheel --version-suffix-for-pypi dev0
- ${{ inputs.chicken-egg-providers }}
+ ${CHICKEN_EGG_PROVIDERS}
if: inputs.chicken-egg-providers != ''
- name: "PyPI constraints"
shell: bash
timeout-minutes: 25
+ env:
+ CHICKEN_EGG_PROVIDERS: ${{ inputs.chicken-egg-providers }}
run: >
breeze release-management generate-constraints --run-in-parallel
--airflow-constraints-mode constraints --answer yes
- --chicken-egg-providers "${{ inputs.chicken-egg-providers }}" --parallelism 3
+ --chicken-egg-providers "${CHICKEN_EGG_PROVIDERS}" --parallelism 3
- name: "Dependency upgrade summary"
shell: bash
+ env:
+ PYTHON_VERSIONS: ${{ env.PYTHON_VERSIONS }}
run: |
- for PYTHON_VERSION in ${{ env.PYTHON_VERSIONS }}; do
+ for PYTHON_VERSION in $PYTHON_VERSIONS; do
echo "Summarizing Python $PYTHON_VERSION"
cat "files/constraints-${PYTHON_VERSION}"/*.md >> $GITHUB_STEP_SUMMARY || true
done
diff --git a/.github/workflows/helm-tests.yml b/.github/workflows/helm-tests.yml
index 8b26769ff4b..9dc300c61c0 100644
--- a/.github/workflows/helm-tests.yml
+++ b/.github/workflows/helm-tests.yml
@@ -32,14 +32,14 @@ on: # yamllint disable-line rule:truthy
description: "Stringified JSON array of helm test packages to test"
required: true
type: string
- image-tag:
- description: "Tag to set for the image"
- required: true
- type: string
default-python-version:
description: "Which version of python should be used by default"
required: true
type: string
+ use-uv:
+ description: "Whether to use uvloop (true/false)"
+ required: true
+ type: string
jobs:
tests-helm:
timeout-minutes: 80
@@ -57,7 +57,6 @@ jobs:
DB_RESET: "false"
JOB_ID: "helm-tests"
USE_XDIST: "true"
- IMAGE_TAG: "${{ inputs.image-tag }}"
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_USERNAME: ${{ github.actor }}
@@ -70,12 +69,16 @@ jobs:
uses: actions/checkout@v4
with:
persist-credentials: false
- - name: "Cleanup docker"
- run: ./scripts/ci/cleanup_docker.sh
- - name: "Prepare breeze & CI image: ${{inputs.default-python-version}}:${{inputs.image-tag}}"
+ - name: "Prepare breeze & CI image: ${{ inputs.default-python-version }}"
uses: ./.github/actions/prepare_breeze_and_image
+ with:
+ platform: "linux/amd64"
+ python: ${{ inputs.default-python-version }}
+ use-uv: ${{ inputs.use-uv }}
- name: "Helm Unit Tests: ${{ matrix.helm-test-package }}"
- run: breeze testing helm-tests --helm-test-package "${{ matrix.helm-test-package }}"
+ env:
+ HELM_TEST_PACKAGE: "${{ matrix.helm-test-package }}"
+ run: breeze testing helm-tests --test-type "${HELM_TEST_PACKAGE}"
tests-helm-release:
timeout-minutes: 80
@@ -95,6 +98,8 @@ jobs:
run: ./scripts/ci/cleanup_docker.sh
- name: "Install Breeze"
uses: ./.github/actions/breeze
+ with:
+ use-uv: ${{ inputs.use-uv }}
- name: Setup git for tagging
run: |
git config --global user.email "name@example.com"
diff --git a/.github/workflows/integration-system-tests.yml b/.github/workflows/integration-system-tests.yml
new file mode 100644
index 00000000000..f992b726e30
--- /dev/null
+++ b/.github/workflows/integration-system-tests.yml
@@ -0,0 +1,207 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: Integration and system tests
+on: # yamllint disable-line rule:truthy
+ workflow_call:
+ inputs:
+ runs-on-as-json-public:
+ description: "The array of labels (in json form) determining public runners."
+ required: true
+ type: string
+ testable-core-integrations:
+ description: "The list of testable core integrations as JSON array."
+ required: true
+ type: string
+ testable-providers-integrations:
+ description: "The list of testable providers integrations as JSON array."
+ required: true
+ type: string
+ run-system-tests:
+ description: "Run system tests (true/false)"
+ required: true
+ type: string
+ default-postgres-version:
+ description: "Default version of Postgres to use"
+ required: true
+ type: string
+ default-mysql-version:
+ description: "Default version of MySQL to use"
+ required: true
+ type: string
+ skip-providers-tests:
+ description: "Skip provider tests (true/false)"
+ required: true
+ type: string
+ run-coverage:
+ description: "Run coverage (true/false)"
+ required: true
+ type: string
+ default-python-version:
+ description: "Which version of python should be used by default"
+ required: true
+ type: string
+ debug-resources:
+ description: "Debug resources (true/false)"
+ required: true
+ type: string
+ use-uv:
+ description: "Whether to use uv"
+ required: true
+ type: string
+jobs:
+ tests-core-integration:
+ timeout-minutes: 130
+ if: inputs.testable-core-integrations != '[]'
+ name: "Integration core ${{ matrix.integration }}"
+ runs-on: ${{ fromJSON(inputs.runs-on-as-json-public) }}
+ strategy:
+ fail-fast: false
+ matrix:
+ integration: ${{ fromJSON(inputs.testable-core-integrations) }}
+ env:
+ BACKEND: "postgres"
+ BACKEND_VERSION: ${{ inputs.default-postgres-version }}"
+ PYTHON_MAJOR_MINOR_VERSION: "${{ inputs.default-python-version }}"
+ JOB_ID: "integration-core-${{ matrix.integration }}"
+ SKIP_PROVIDERS_TESTS: "${{ inputs.skip-providers-tests }}"
+ ENABLE_COVERAGE: "${{ inputs.run-coverage}}"
+ DEBUG_RESOURCES: "${{ inputs.debug-resources }}"
+ GITHUB_REPOSITORY: ${{ github.repository }}
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ GITHUB_USERNAME: ${{ github.actor }}
+ VERBOSE: "true"
+ steps:
+ - name: "Cleanup repo"
+ shell: bash
+ run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ uses: actions/checkout@v4
+ with:
+ persist-credentials: false
+ - name: "Prepare breeze & CI image: ${{ inputs.default-python-version }}"
+ uses: ./.github/actions/prepare_breeze_and_image
+ with:
+ platform: "linux/amd64"
+ python: ${{ inputs.default-python-version }}
+ use-uv: ${{ inputs.use-uv }}
+ - name: "Integration: core ${{ matrix.integration }}"
+ env:
+ INTEGRATION: "${{ matrix.integration }}"
+ # yamllint disable rule:line-length
+ run: ./scripts/ci/testing/run_integration_tests_with_retry.sh core "${INTEGRATION}"
+ - name: "Post Tests success"
+ uses: ./.github/actions/post_tests_success
+ with:
+ codecov-token: ${{ secrets.CODECOV_TOKEN }}
+ python-version: ${{ inputs.default-python-version }}
+ - name: "Post Tests failure"
+ uses: ./.github/actions/post_tests_failure
+ if: failure()
+
+ tests-providers-integration:
+ timeout-minutes: 130
+ if: inputs.testable-providers-integrations != '[]' && inputs.skip-providers-tests != 'true'
+ name: "Integration: providers ${{ matrix.integration }}"
+ runs-on: ${{ fromJSON(inputs.runs-on-as-json-public) }}
+ strategy:
+ fail-fast: false
+ matrix:
+ integration: ${{ fromJSON(inputs.testable-providers-integrations) }}
+ env:
+ BACKEND: "postgres"
+ BACKEND_VERSION: ${{ inputs.default-postgres-version }}"
+ PYTHON_MAJOR_MINOR_VERSION: "${{ inputs.default-python-version }}"
+ JOB_ID: "integration-providers-${{ matrix.integration }}"
+ SKIP_PROVIDERS_TESTS: "${{ inputs.skip-providers-tests }}"
+ ENABLE_COVERAGE: "${{ inputs.run-coverage}}"
+ DEBUG_RESOURCES: "${{ inputs.debug-resources }}"
+ GITHUB_REPOSITORY: ${{ github.repository }}
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ GITHUB_USERNAME: ${{ github.actor }}
+ VERBOSE: "true"
+ steps:
+ - name: "Cleanup repo"
+ shell: bash
+ run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ uses: actions/checkout@v4
+ with:
+ persist-credentials: false
+ - name: "Prepare breeze & CI image: ${{ inputs.default-python-version }}"
+ uses: ./.github/actions/prepare_breeze_and_image
+ with:
+ platform: "linux/amd64"
+ python: ${{ inputs.default-python-version }}
+ use-uv: ${{ inputs.use-uv }}
+ - name: "Integration: providers ${{ matrix.integration }}"
+ env:
+ INTEGRATION: "${{ matrix.integration }}"
+ run: ./scripts/ci/testing/run_integration_tests_with_retry.sh providers "${INTEGRATION}"
+ - name: "Post Tests success"
+ uses: ./.github/actions/post_tests_success
+ with:
+ codecov-token: ${{ secrets.CODECOV_TOKEN }}
+ python-version: ${{ inputs.default-python-version }}
+ - name: "Post Tests failure"
+ uses: ./.github/actions/post_tests_failure
+ if: failure()
+
+ tests-system:
+ timeout-minutes: 130
+ if: inputs.run-system-tests == 'true'
+ name: "System Tests"
+ runs-on: ${{ fromJSON(inputs.runs-on-as-json-public) }}
+ env:
+ BACKEND: "postgres"
+ BACKEND_VERSION: ${{ inputs.default-postgres-version }}"
+ PYTHON_MAJOR_MINOR_VERSION: "${{ inputs.default-python-version }}"
+ JOB_ID: "system"
+ SKIP_PROVIDERS_TESTS: "${{ inputs.skip-providers-tests }}"
+ ENABLE_COVERAGE: "${{ inputs.run-coverage}}"
+ DEBUG_RESOURCES: "${{ inputs.debug-resources }}"
+ GITHUB_REPOSITORY: ${{ github.repository }}
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ GITHUB_USERNAME: ${{ github.actor }}
+ VERBOSE: "true"
+ steps:
+ - name: "Cleanup repo"
+ shell: bash
+ run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ uses: actions/checkout@v4
+ with:
+ persist-credentials: false
+ - name: "Prepare breeze & CI image: ${{ inputs.default-python-version }}"
+ uses: ./.github/actions/prepare_breeze_and_image
+ with:
+ platform: "linux/amd64"
+ python: ${{ inputs.default-python-version }}
+ use-uv: ${{ inputs.use-uv }}
+ - name: "System Tests"
+ run: >
+ ./scripts/ci/testing/run_system_tests.sh
+ tests/system/example_empty.py providers/tests/system/example_empty.py
+ - name: "Post Tests success"
+ uses: ./.github/actions/post_tests_success
+ with:
+ codecov-token: ${{ secrets.CODECOV_TOKEN }}
+ python-version: ${{ inputs.default-python-version }}
+ - name: "Post Tests failure"
+ uses: ./.github/actions/post_tests_failure
+ if: failure()
diff --git a/.github/workflows/integration-tests.yml b/.github/workflows/integration-tests.yml
deleted file mode 100644
index 530d0f9fc56..00000000000
--- a/.github/workflows/integration-tests.yml
+++ /dev/null
@@ -1,103 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
----
-name: Integration tests
-on: # yamllint disable-line rule:truthy
- workflow_call:
- inputs:
- runs-on-as-json-public:
- description: "The array of labels (in json form) determining public runners."
- required: true
- type: string
- image-tag:
- description: "Tag to set for the image"
- required: true
- type: string
- testable-integrations:
- description: "The list of testable integrations as JSON array."
- required: true
- type: string
- default-postgres-version:
- description: "Default version of Postgres to use"
- required: true
- type: string
- default-mysql-version:
- description: "Default version of MySQL to use"
- required: true
- type: string
- skip-provider-tests:
- description: "Skip provider tests (true/false)"
- required: true
- type: string
- run-coverage:
- description: "Run coverage (true/false)"
- required: true
- type: string
- default-python-version:
- description: "Which version of python should be used by default"
- required: true
- type: string
- debug-resources:
- description: "Debug resources (true/false)"
- required: true
- type: string
-jobs:
- tests-integration:
- timeout-minutes: 130
- if: inputs.testable-integrations != '[]'
- name: "Integration Tests: ${{ matrix.integration }}"
- runs-on: ${{ fromJSON(inputs.runs-on-as-json-public) }}
- strategy:
- fail-fast: false
- matrix:
- integration: ${{ fromJSON(inputs.testable-integrations) }}
- env:
- IMAGE_TAG: "${{ inputs.image-tag }}"
- BACKEND: "postgres"
- BACKEND_VERSION: ${{ inputs.default-postgres-version }}"
- PYTHON_MAJOR_MINOR_VERSION: "${{ inputs.default-python-version }}"
- JOB_ID: "integration-${{ matrix.integration }}"
- SKIP_PROVIDER_TESTS: "${{ inputs.skip-provider-tests }}"
- ENABLE_COVERAGE: "${{ inputs.run-coverage}}"
- DEBUG_RESOURCES: "${{ inputs.debug-resources }}"
- GITHUB_REPOSITORY: ${{ github.repository }}
- GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- GITHUB_USERNAME: ${{ github.actor }}
- VERBOSE: "true"
- steps:
- - name: "Cleanup repo"
- shell: bash
- run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
- uses: actions/checkout@v4
- with:
- persist-credentials: false
- - name: "Cleanup docker"
- run: ./scripts/ci/cleanup_docker.sh
- - name: "Prepare breeze & CI image: ${{ inputs.default-python-version }}:${{ inputs.image-tag }}"
- uses: ./.github/actions/prepare_breeze_and_image
- - name: "Integration Tests: ${{ matrix.integration }}"
- run: ./scripts/ci/testing/run_integration_tests_with_retry.sh ${{ matrix.integration }}
- - name: "Post Tests success: Integration Tests ${{ matrix.integration }}"
- uses: ./.github/actions/post_tests_success
- with:
- codecov-token: ${{ secrets.CODECOV_TOKEN }}
- python-version: ${{ inputs.default-python-version }}
- - name: "Post Tests failure: Integration Tests ${{ matrix.integration }}"
- uses: ./.github/actions/post_tests_failure
- if: failure()
diff --git a/.github/workflows/k8s-tests.yml b/.github/workflows/k8s-tests.yml
index 3b3e067038d..6f867af65e9 100644
--- a/.github/workflows/k8s-tests.yml
+++ b/.github/workflows/k8s-tests.yml
@@ -20,24 +20,20 @@ name: K8s tests
on: # yamllint disable-line rule:truthy
workflow_call:
inputs:
- runs-on-as-json-default:
- description: "The array of labels (in json form) determining default runner used for the build."
+ platform:
+ description: "Platform for the build - 'linux/amd64' or 'linux/arm64'"
required: true
type: string
- image-tag:
- description: "Tag to set for the image"
+ runs-on-as-json-default:
+ description: "The array of labels (in json form) determining default runner used for the build."
required: true
type: string
python-versions-list-as-string:
description: "List of Python versions to test: space separated string"
required: true
type: string
- kubernetes-versions-list-as-string:
- description: "List of Kubernetes versions to test"
- required: true
- type: string
- kubernetes-combos-list-as-string:
- description: "List of combinations of Kubernetes and Python versions to test: space separated string"
+ kubernetes-combos:
+ description: "Array of combinations of Kubernetes and Python versions to test"
required: true
type: string
include-success-outputs:
@@ -54,20 +50,18 @@ on: # yamllint disable-line rule:truthy
type: string
jobs:
tests-kubernetes:
- timeout-minutes: 240
- name: "\
- K8S System:${{ matrix.executor }} - ${{ matrix.use-standard-naming }} - \
- ${{ inputs.kubernetes-versions-list-as-string }}"
+ timeout-minutes: 60
+ name: "K8S System:${{ matrix.executor }}-${{ matrix.kubernetes-combo }}-${{ matrix.use-standard-naming }}"
runs-on: ${{ fromJSON(inputs.runs-on-as-json-default) }}
strategy:
matrix:
executor: [KubernetesExecutor, CeleryExecutor, LocalExecutor]
use-standard-naming: [true, false]
+ kubernetes-combo: ${{ fromJSON(inputs.kubernetes-combos) }}
fail-fast: false
env:
DEBUG_RESOURCES: ${{ inputs.debug-resources }}
INCLUDE_SUCCESS_OUTPUTS: ${{ inputs.include-success-outputs }}
- IMAGE_TAG: ${{ inputs.image-tag }}
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_USERNAME: ${{ github.actor }}
@@ -76,55 +70,58 @@ jobs:
- name: "Cleanup repo"
shell: bash
run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ - name: "Prepare PYTHON_MAJOR_MINOR_VERSION and KUBERNETES_VERSION"
+ id: prepare-versions
+ env:
+ KUBERNETES_COMBO: ${{ matrix.kubernetes-combo }}
+ run: |
+ echo "PYTHON_MAJOR_MINOR_VERSION=${KUBERNETES_COMBO}" | sed 's/-.*//' >> $GITHUB_ENV
+ echo "KUBERNETES_VERSION=${KUBERNETES_COMBO}" | sed 's/=[^-]*-/=/' >> $GITHUB_ENV
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v4
with:
persist-credentials: false
- - name: "Cleanup docker"
- run: ./scripts/ci/cleanup_docker.sh
- - name: "Install Breeze"
- uses: ./.github/actions/breeze
- id: breeze
- - name: Login to ghcr.io
- run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
- - name: Pull PROD images ${{ inputs.python-versions-list-as-string }}:${{ inputs.image-tag }}
- run: breeze prod-image pull --run-in-parallel --tag-as-latest
- env:
- PYTHON_VERSIONS: ${{ inputs.python-versions-list-as-string }}
- # Force more parallelism for pull even on public images
- PARALLELISM: 6
- - name: "Cache bin folder with tools for kubernetes testing"
- uses: actions/cache@v4
+ # env.PYTHON_MAJOR_MINOR_VERSION, env.KUBERNETES_VERSION are set in the previous
+ # step id: prepare-versions
+ - name: "Prepare breeze & PROD image: ${{ env.PYTHON_MAJOR_MINOR_VERSION }}"
+ uses: ./.github/actions/prepare_breeze_and_image
with:
- path: ".build/.k8s-env"
- key: "\
- k8s-env-${{ steps.breeze.outputs.host-python-version }}-\
- ${{ hashFiles('scripts/ci/kubernetes/k8s_requirements.txt','hatch_build.py') }}"
- - name: "Switch breeze to use uv"
- run: breeze setup config --use-uv
- if: inputs.use-uv == 'true'
- - name: Run complete K8S tests ${{ inputs.kubernetes-combos-list-as-string }}
- run: breeze k8s run-complete-tests --run-in-parallel --upgrade --no-copy-local-sources
+ platform: ${{ inputs.platform }}
+ image-type: "prod"
+ python: ${{ env.PYTHON_MAJOR_MINOR_VERSION }}
+ use-uv: ${{ inputs.use-uv }}
+ id: breeze
+ # preparing k8s environment with uv takes < 15 seconds with `uv` - there is no point in caching it.
+ - name: "\
+ Run complete K8S tests ${{ matrix.executor }}-${{ env.PYTHON_MAJOR_MINOR_VERSION }}-\
+ ${{env.KUBERNETES_VERSION}}-${{ matrix.use-standard-naming }}"
+ run: breeze k8s run-complete-tests --upgrade --no-copy-local-sources
env:
- PYTHON_VERSIONS: ${{ inputs.python-versions-list-as-string }}
- KUBERNETES_VERSIONS: ${{ inputs.kubernetes-versions-list-as-string }}
EXECUTOR: ${{ matrix.executor }}
USE_STANDARD_NAMING: ${{ matrix.use-standard-naming }}
VERBOSE: "false"
- - name: Upload KinD logs on failure ${{ inputs.kubernetes-combos-list-as-string }}
+ - name: "\
+ Upload KinD logs on failure ${{ matrix.executor }}-${{ matrix.kubernetes-combo }}-\
+ ${{ matrix.use-standard-naming }}"
uses: actions/upload-artifact@v4
if: failure() || cancelled()
with:
- name: kind-logs-${{ matrix.executor }}-${{ matrix.use-standard-naming }}
+ name: "\
+ kind-logs-${{ matrix.kubernetes-combo }}-${{ matrix.executor }}-\
+ ${{ matrix.use-standard-naming }}"
path: /tmp/kind_logs_*
- retention-days: 7
- - name: Upload test resource logs on failure ${{ inputs.kubernetes-combos-list-as-string }}
+ retention-days: '7'
+ - name: "\
+ Upload test resource logs on failure ${{ matrix.executor }}-${{ matrix.kubernetes-combo }}-\
+ ${{ matrix.use-standard-naming }}"
uses: actions/upload-artifact@v4
if: failure() || cancelled()
with:
- name: k8s-test-resources-${{ matrix.executor }}-${{ matrix.use-standard-naming }}
+ name: "\
+ k8s-test-resources-${{ matrix.kubernetes-combo }}-${{ matrix.executor }}-\
+ ${{ matrix.use-standard-naming }}"
path: /tmp/k8s_test_resources_*
- retention-days: 7
+ retention-days: '7'
- name: "Delete clusters just in case they are left"
run: breeze k8s delete-cluster --all
if: always()
diff --git a/.github/workflows/news-fragment.yml b/.github/workflows/news-fragment.yml
new file mode 100644
index 00000000000..73e58a01937
--- /dev/null
+++ b/.github/workflows/news-fragment.yml
@@ -0,0 +1,81 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: CI
+
+on: # yamllint disable-line rule:truthy
+ pull_request:
+ types: [labeled, unlabeled, opened, reopened, synchronize]
+
+jobs:
+ check-news-fragment:
+ name: Check News Fragment
+ runs-on: ubuntu-20.04
+ if: "contains(github.event.pull_request.labels.*.name, 'airflow3.0:breaking')"
+
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ persist-credentials: false
+ # `towncrier check` runs `git diff --name-only origin/main...`, which
+ # needs a non-shallow clone.
+ fetch-depth: 0
+
+ - name: Check news fragment existence
+ env:
+ BASE_REF: ${{ github.base_ref }}
+ run: >
+ python -m pip install --upgrade uv &&
+ uv tool run towncrier check
+ --dir .
+ --config newsfragments/config.toml
+ --compare-with origin/${BASE_REF}
+ ||
+ {
+ printf "\033[1;33mMissing significant newsfragment for PR labeled with
+ 'airflow3.0:breaking'.\nCheck
+ https://github.com/apache/airflow/blob/main/contributing-docs/16_contribution_workflow.rst
+ for guidance.\033[m\n"
+ &&
+ false
+ ; }
+
+ - name: Check news fragment contains change types
+ env:
+ BASE_REF: ${{ github.base_ref }}
+ run: >
+ change_types=(
+ 'DAG changes'
+ 'Config changes'
+ 'API changes'
+ 'CLI changes'
+ 'Behaviour changes'
+ 'Plugin changes'
+ 'Dependency change'
+ )
+ news_fragment_content=`git diff origin/${BASE_REF} newsfragments/*.significant.rst`
+
+ for type in "${change_types[@]}"; do
+ if [[ $news_fragment_content != *"$type"* ]]; then
+ printf "\033[1;33mMissing change type '$type' in significant newsfragment for PR labeled with
+ 'airflow3.0:breaking'.\nCheck
+ https://github.com/apache/airflow/blob/main/contributing-docs/16_contribution_workflow.rst
+ for guidance.\033[m\n"
+ exit 1
+ fi
+ done
diff --git a/.github/workflows/prod-image-build.yml b/.github/workflows/prod-image-build.yml
index 75d9d0054ec..d90d1910f93 100644
--- a/.github/workflows/prod-image-build.yml
+++ b/.github/workflows/prod-image-build.yml
@@ -30,13 +30,6 @@ on: # yamllint disable-line rule:truthy
variations.
required: true
type: string
- do-build:
- description: >
- Whether to actually do the build (true/false). If set to false, the build is done
- already in pull-request-target workflow, so we skip it here.
- required: false
- default: "true"
- type: string
upload-package-artifact:
description: >
Whether to upload package artifacts (true/false). If false, the job will rely on artifacts prepared
@@ -62,6 +55,11 @@ on: # yamllint disable-line rule:truthy
description: "Whether to push image to the registry (true/false)"
required: true
type: string
+ upload-image-artifact:
+ description: "Whether to upload docker image artifact"
+ required: false
+ default: "false"
+ type: string
debian-version:
description: "Base Debian distribution to use for the build (bookworm)"
type: string
@@ -74,10 +72,6 @@ on: # yamllint disable-line rule:truthy
description: "Whether to use uv to build the image (true/false)"
required: true
type: string
- image-tag:
- description: "Tag to set for the image"
- required: true
- type: string
python-versions:
description: "JSON-formatted array of Python versions to build images from"
required: true
@@ -114,12 +108,21 @@ on: # yamllint disable-line rule:truthy
description: "Docker cache specification to build the image (registry, local, disabled)."
required: true
type: string
+ disable-airflow-repo-cache:
+ description: "Disable airflow repo cache read from main."
+ required: true
+ type: string
+ prod-image-build:
+ description: "Whether this is a prod-image build (true/false)"
+ required: true
+ type: string
jobs:
build-prod-packages:
- name: "${{ inputs.do-build == 'true' && 'Build' || 'Skip building' }} Airflow and provider packages"
+ name: "Build Airflow and provider packages"
timeout-minutes: 10
runs-on: ${{ fromJSON(inputs.runs-on-as-json-public) }}
+ if: inputs.prod-image-build == 'true'
env:
PYTHON_MAJOR_MINOR_VERSION: "${{ inputs.default-python-version }}"
VERSION_SUFFIX_FOR_PYPI: ${{ inputs.branch == 'main' && 'dev0' || '' }}
@@ -127,32 +130,23 @@ jobs:
- name: "Cleanup repo"
shell: bash
run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
- if: inputs.do-build == 'true' && inputs.upload-package-artifact == 'true'
+ if: inputs.upload-package-artifact == 'true'
- name: "Checkout target branch"
uses: actions/checkout@v4
with:
persist-credentials: false
- - name: "Checkout target commit"
- uses: ./.github/actions/checkout_target_commit
- with:
- target-commit-sha: ${{ inputs.target-commit-sha }}
- pull-request-target: ${{ inputs.pull-request-target }}
- is-committer-build: ${{ inputs.is-committer-build }}
- if: inputs.do-build == 'true' && inputs.upload-package-artifact == 'true'
- name: "Cleanup docker"
run: ./scripts/ci/cleanup_docker.sh
- if: inputs.do-build == 'true' && inputs.upload-package-artifact == 'true'
- - uses: actions/setup-python@v5
- with:
- python-version: "${{ inputs.default-python-version }}"
- if: inputs.do-build == 'true' && inputs.upload-package-artifact == 'true'
+ if: inputs.upload-package-artifact == 'true'
- name: "Cleanup dist and context file"
shell: bash
run: rm -fv ./dist/* ./docker-context-files/*
- if: inputs.do-build == 'true' && inputs.upload-package-artifact == 'true'
+ if: inputs.upload-package-artifact == 'true'
- name: "Install Breeze"
uses: ./.github/actions/breeze
- if: inputs.do-build == 'true' && inputs.upload-package-artifact == 'true'
+ with:
+ use-uv: ${{ inputs.use-uv }}
+ if: inputs.upload-package-artifact == 'true'
- name: "Prepare providers packages"
shell: bash
run: >
@@ -160,23 +154,28 @@ jobs:
--package-list-file ./prod_image_installed_providers.txt
--package-format wheel
if: >
- inputs.do-build == 'true' &&
inputs.upload-package-artifact == 'true' &&
inputs.build-provider-packages == 'true'
- name: "Prepare chicken-eggs provider packages"
shell: bash
+ env:
+ CHICKEN_EGG_PROVIDERS: ${{ inputs.chicken-egg-providers }}
run: >
breeze release-management prepare-provider-packages
- --package-format wheel ${{ inputs.chicken-egg-providers }}
+ --package-format wheel ${CHICKEN_EGG_PROVIDERS}
if: >
- inputs.do-build == 'true' &&
inputs.upload-package-artifact == 'true' &&
inputs.chicken-egg-providers != ''
- name: "Prepare airflow package"
shell: bash
run: >
breeze release-management prepare-airflow-package --package-format wheel
- if: inputs.do-build == 'true' && inputs.upload-package-artifact == 'true'
+ if: inputs.upload-package-artifact == 'true'
+ - name: "Prepare task-sdk package"
+ shell: bash
+ run: >
+ breeze release-management prepare-task-sdk-package --package-format wheel
+ if: inputs.upload-package-artifact == 'true'
- name: "Upload prepared packages as artifacts"
uses: actions/upload-artifact@v4
with:
@@ -184,25 +183,21 @@ jobs:
path: ./dist
retention-days: 7
if-no-files-found: error
- if: inputs.do-build == 'true' && inputs.upload-package-artifact == 'true'
+ if: inputs.upload-package-artifact == 'true'
build-prod-images:
strategy:
fail-fast: false
matrix:
- # yamllint disable-line rule:line-length
- python-version: ${{ inputs.do-build == 'true' && fromJSON(inputs.python-versions) || fromJSON('[""]') }}
+ python-version: ${{ fromJSON(inputs.python-versions) || fromJSON('[""]') }}
timeout-minutes: 80
- name: "\
-${{ inputs.do-build == 'true' && 'Build' || 'Skip building' }} \
-PROD ${{ inputs.build-type }} image\
-${{ matrix.python-version }}${{ inputs.do-build == 'true' && ':' || '' }}\
-${{ inputs.do-build == 'true' && inputs.image-tag || '' }}"
+ name: "Build PROD ${{ inputs.build-type }} image ${{ matrix.python-version }}"
runs-on: ${{ fromJSON(inputs.runs-on-as-json-public) }}
needs:
- build-prod-packages
env:
BACKEND: sqlite
+ PYTHON_MAJOR_MINOR_VERSION: "${{ matrix.python-version }}"
DEFAULT_BRANCH: ${{ inputs.branch }}
DEFAULT_CONSTRAINTS_BRANCH: ${{ inputs.constraints-branch }}
VERSION_SUFFIX_FOR_PYPI: ${{ inputs.branch == 'main' && 'dev0' || '' }}
@@ -216,88 +211,88 @@ ${{ inputs.do-build == 'true' && inputs.image-tag || '' }}"
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_USERNAME: ${{ github.actor }}
- USE_UV: ${{ inputs.use-uv }}
VERBOSE: "true"
steps:
- name: "Cleanup repo"
shell: bash
run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
- if: inputs.do-build == 'true'
- name: "Checkout target branch"
uses: actions/checkout@v4
with:
persist-credentials: false
- - name: "Checkout target commit"
- uses: ./.github/actions/checkout_target_commit
- with:
- target-commit-sha: ${{ inputs.target-commit-sha }}
- pull-request-target: ${{ inputs.pull-request-target }}
- is-committer-build: ${{ inputs.is-committer-build }}
- if: inputs.do-build == 'true'
- name: "Cleanup docker"
run: ./scripts/ci/cleanup_docker.sh
- if: inputs.do-build == 'true'
- name: "Install Breeze"
uses: ./.github/actions/breeze
- if: inputs.do-build == 'true'
- - name: "Regenerate dependencies in case they was modified manually so that we can build an image"
- shell: bash
- run: |
- pip install rich>=12.4.4 pyyaml
- python scripts/ci/pre_commit/update_providers_dependencies.py
- if: inputs.do-build == 'true' && inputs.upgrade-to-newer-dependencies != 'false'
+ with:
+ use-uv: ${{ inputs.use-uv }}
- name: "Cleanup dist and context file"
shell: bash
run: rm -fv ./dist/* ./docker-context-files/*
- if: inputs.do-build == 'true'
- name: "Download packages prepared as artifacts"
uses: actions/download-artifact@v4
with:
name: prod-packages
path: ./docker-context-files
- if: inputs.do-build == 'true'
- name: "Download constraints"
uses: actions/download-artifact@v4
with:
name: constraints
path: ./docker-context-files
- if: inputs.do-build == 'true'
- - name: Login to ghcr.io
- shell: bash
- run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
- if: inputs.do-build == 'true'
- - name: "Build PROD images w/ source providers ${{ matrix.python-version }}:${{ inputs.image-tag }}"
+ - name: "Login to ghcr.io"
+ env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ ACTOR: ${{ github.actor }}
+ run: echo "${GITHUB_TOKEN}" | docker login ghcr.io -u ${ACTOR} --password-stdin
+ - name: "Build PROD images w/ source providers ${{ env.PYTHON_MAJOR_MINOR_VERSION }}"
shell: bash
run: >
- breeze prod-image build --tag-as-latest --image-tag "${{ inputs.image-tag }}"
+ breeze prod-image build
+ --builder airflow_cache
--commit-sha "${{ github.sha }}"
- --install-packages-from-context --airflow-constraints-mode constraints-source-providers
- --use-constraints-for-context-packages --python "${{ matrix.python-version }}"
+ --install-packages-from-context
+ --airflow-constraints-mode constraints-source-providers
+ --use-constraints-for-context-packages
env:
PUSH: ${{ inputs.push-image }}
DOCKER_CACHE: ${{ inputs.docker-cache }}
+ DISABLE_AIRFLOW_REPO_CACHE: ${{ inputs.disable-airflow-repo-cache }}
DEBIAN_VERSION: ${{ inputs.debian-version }}
INSTALL_MYSQL_CLIENT_TYPE: ${{ inputs.install-mysql-client-type }}
UPGRADE_TO_NEWER_DEPENDENCIES: ${{ inputs.upgrade-to-newer-dependencies }}
INCLUDE_NOT_READY_PROVIDERS: "true"
- if: inputs.do-build == 'true' && inputs.build-provider-packages == 'true'
- - name: "Build PROD images with PyPi providers ${{ matrix.python-version }}:${{ inputs.image-tag }}"
+ if: inputs.build-provider-packages == 'true'
+ - name: "Build PROD images with PyPi providers ${{ env.PYTHON_MAJOR_MINOR_VERSION }}"
shell: bash
run: >
- breeze prod-image build --builder airflow_cache --tag-as-latest
- --image-tag "${{ inputs.image-tag }}" --commit-sha "${{ github.sha }}"
- --install-packages-from-context --airflow-constraints-mode constraints
- --use-constraints-for-context-packages --python "${{ matrix.python-version }}"
+ breeze prod-image build
+ --builder airflow_cache
+ --commit-sha "${{ github.sha }}"
+ --install-packages-from-context
+ --airflow-constraints-mode constraints
+ --use-constraints-for-context-packages
env:
PUSH: ${{ inputs.push-image }}
DOCKER_CACHE: ${{ inputs.docker-cache }}
+ DISABLE_AIRFLOW_REPO_CACHE: ${{ inputs.disable-airflow-repo-cache }}
DEBIAN_VERSION: ${{ inputs.debian-version }}
INSTALL_MYSQL_CLIENT_TYPE: ${{ inputs.install-mysql-client-type }}
UPGRADE_TO_NEWER_DEPENDENCIES: ${{ inputs.upgrade-to-newer-dependencies }}
INCLUDE_NOT_READY_PROVIDERS: "true"
- if: inputs.do-build == 'true' && inputs.build-provider-packages != 'true'
- - name: Verify PROD image ${{ matrix.python-version }}:${{ inputs.image-tag }}
+ if: inputs.build-provider-packages != 'true'
+ - name: "Verify PROD image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}"
+ run: breeze prod-image verify
+ - name: "Export PROD docker image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}"
+ env:
+ PLATFORM: ${{ inputs.platform }}
run: >
- breeze prod-image verify --image-tag "${{ inputs.image-tag }}"
- --python "${{ matrix.python-version }}"
- if: inputs.do-build == 'true'
+ breeze prod-image save --platform "${PLATFORM}"
+ if: inputs.upload-image-artifact == 'true'
+ - name: "Stash PROD docker image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}"
+ uses: apache/infrastructure-actions/stash/save@c94b890bbedc2fc61466d28e6bd9966bc6c6643c
+ with:
+ key: prod-image-save-${{ inputs.platform }}-${{ env.PYTHON_MAJOR_MINOR_VERSION }}
+ path: "/tmp/prod-image-save-*-${{ env.PYTHON_MAJOR_MINOR_VERSION }}.tar"
+ if-no-files-found: 'error'
+ retention-days: '2'
+ if: inputs.upload-image-artifact == 'true'
diff --git a/.github/workflows/prod-image-extra-checks.yml b/.github/workflows/prod-image-extra-checks.yml
index 82d327ba2f1..f5a4b771436 100644
--- a/.github/workflows/prod-image-extra-checks.yml
+++ b/.github/workflows/prod-image-extra-checks.yml
@@ -40,9 +40,6 @@ on: # yamllint disable-line rule:truthy
description: "Whether to use uv to build the image (true/false)"
required: true
type: string
- image-tag:
- required: true
- type: string
build-provider-packages:
description: "Whether to build provider packages (true/false). If false providers are from PyPI"
required: true
@@ -63,14 +60,18 @@ on: # yamllint disable-line rule:truthy
description: "Docker cache specification to build the image (registry, local, disabled)."
required: true
type: string
+ disable-airflow-repo-cache:
+ description: "Disable airflow repo cache read from main."
+ required: true
+ type: string
jobs:
myssql-client-image:
uses: ./.github/workflows/prod-image-build.yml
with:
runs-on-as-json-public: ${{ inputs.runs-on-as-json-public }}
build-type: "MySQL Client"
+ upload-image-artifact: "false"
upload-package-artifact: "false"
- image-tag: mysql-${{ inputs.image-tag }}
install-mysql-client-type: "mysql"
python-versions: ${{ inputs.python-versions }}
default-python-version: ${{ inputs.default-python-version }}
@@ -84,6 +85,8 @@ jobs:
chicken-egg-providers: ${{ inputs.chicken-egg-providers }}
constraints-branch: ${{ inputs.constraints-branch }}
docker-cache: ${{ inputs.docker-cache }}
+ disable-airflow-repo-cache: ${{ inputs.disable-airflow-repo-cache }}
+ prod-image-build: "true"
pip-image:
uses: ./.github/workflows/prod-image-build.yml
@@ -92,8 +95,8 @@ jobs:
with:
runs-on-as-json-public: ${{ inputs.runs-on-as-json-public }}
build-type: "pip"
+ upload-image-artifact: "false"
upload-package-artifact: "false"
- image-tag: mysql-${{ inputs.image-tag }}
install-mysql-client-type: "mysql"
python-versions: ${{ inputs.python-versions }}
default-python-version: ${{ inputs.default-python-version }}
@@ -107,3 +110,5 @@ jobs:
chicken-egg-providers: ${{ inputs.chicken-egg-providers }}
constraints-branch: ${{ inputs.constraints-branch }}
docker-cache: ${{ inputs.docker-cache }}
+ disable-airflow-repo-cache: ${{ inputs.disable-airflow-repo-cache }}
+ prod-image-build: "true"
diff --git a/.github/workflows/push-image-cache.yml b/.github/workflows/push-image-cache.yml
index 0dc83a3fd66..b1c9d127542 100644
--- a/.github/workflows/push-image-cache.yml
+++ b/.github/workflows/push-image-cache.yml
@@ -76,6 +76,10 @@ on: # yamllint disable-line rule:truthy
description: "Docker cache specification to build the image (registry, local, disabled)."
required: true
type: string
+ disable-airflow-repo-cache:
+ description: "Disable airflow repo cache read from main."
+ required: true
+ type: string
jobs:
push-ci-image-cache:
name: "Push CI ${{ inputs.cache-type }}:${{ matrix.python }} image cache "
@@ -100,12 +104,13 @@ jobs:
DEFAULT_BRANCH: ${{ inputs.branch }}
DEFAULT_CONSTRAINTS_BRANCH: ${{ inputs.constraints-branch }}
DOCKER_CACHE: ${{ inputs.docker-cache }}
+ DISABLE_AIRFLOW_REPO_CACHE: ${{ inputs.disable-airflow-repo-cache }}
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_USERNAME: ${{ github.actor }}
INCLUDE_SUCCESS_OUTPUTS: "${{ inputs.include-success-outputs }}"
INSTALL_MYSQL_CLIENT_TYPE: ${{ inputs.install-mysql-client-type }}
- USE_UV: ${{ inputs.use-uv }}
+ PYTHON_MAJOR_MINOR_VERSION: "${{ matrix.python }}"
UPGRADE_TO_NEWER_DEPENDENCIES: "false"
VERBOSE: "true"
VERSION_SUFFIX_FOR_PYPI: "dev0"
@@ -121,23 +126,33 @@ jobs:
run: ./scripts/ci/cleanup_docker.sh
- name: "Install Breeze"
uses: ./.github/actions/breeze
- - name: "Start ARM instance"
- run: ./scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh
- if: inputs.platform == 'linux/arm64'
+ with:
+ use-uv: ${{ inputs.use-uv }}
- name: Login to ghcr.io
- run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
- - name: "Push CI ${{ inputs.cache-type }} cache: ${{ matrix.python }} ${{ inputs.platform }}"
- run: >
- breeze ci-image build --builder airflow_cache --prepare-buildx-cache
- --platform "${{ inputs.platform }}" --python ${{ matrix.python }}
- - name: "Stop ARM instance"
- run: ./scripts/ci/images/ci_stop_arm_instance.sh
- if: always() && inputs.platform == 'linux/arm64'
- - name: "Push CI latest images: ${{ matrix.python }} (linux/amd64 only)"
+ env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ ACTOR: ${{ github.actor }}
+ run: echo "${GITHUB_TOKEN}" | docker login ghcr.io -u ${ACTOR} --password-stdin
+ - name: "Push CI latest images: ${{ env.PYTHON_MAJOR_MINOR_VERSION }} (linux/amd64 only)"
+ env:
+ PLATFORM: ${{ inputs.platform }}
run: >
- breeze ci-image build --builder airflow_cache --push
- --python "${{ matrix.python }}" --platform "${{ inputs.platform }}"
+ breeze
+ ci-image build
+ --builder airflow_cache
+ --platform "${PLATFORM}"
+ --push
if: inputs.push-latest-images == 'true' && inputs.platform == 'linux/amd64'
+ # yamllint disable-line rule:line-length
+ - name: "Push CI ${{ inputs.cache-type }} cache:${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ inputs.platform }}"
+ env:
+ PLATFORM: ${{ inputs.platform }}
+ run: >
+ breeze ci-image build
+ --builder airflow_cache
+ --prepare-buildx-cache
+ --platform "${PLATFORM}"
+ --push
push-prod-image-cache:
name: "Push PROD ${{ inputs.cache-type }}:${{ matrix.python }} image cache"
@@ -162,12 +177,13 @@ jobs:
DEFAULT_BRANCH: ${{ inputs.branch }}
DEFAULT_CONSTRAINTS_BRANCH: ${{ inputs.constraints-branch }}
DOCKER_CACHE: ${{ inputs.docker-cache }}
+ DISABLE_AIRFLOW_REPO_CACHE: ${{ inputs.disable-airflow-repo-cache }}
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_USERNAME: ${{ github.actor }}
INSTALL_MYSQL_CLIENT_TYPE: ${{ inputs.install-mysql-client-type }}
+ PYTHON_MAJOR_MINOR_VERSION: "${{ matrix.python }}"
UPGRADE_TO_NEWER_DEPENDENCIES: "false"
- USE_UV: ${{ inputs.branch == 'main' && inputs.use-uv || 'false' }}
VERBOSE: "true"
VERSION_SUFFIX_FOR_PYPI: "dev0"
if: inputs.include-prod-images == 'true'
@@ -183,6 +199,8 @@ jobs:
run: ./scripts/ci/cleanup_docker.sh
- name: "Install Breeze"
uses: ./.github/actions/breeze
+ with:
+ use-uv: ${{ inputs.use-uv }}
- name: "Cleanup dist and context file"
run: rm -fv ./dist/* ./docker-context-files/*
- name: "Download packages prepared as artifacts"
@@ -190,25 +208,33 @@ jobs:
with:
name: prod-packages
path: ./docker-context-files
- - name: "Start ARM instance"
- run: ./scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh
- if: inputs.platform == 'linux/arm64'
- name: Login to ghcr.io
- run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
- - name: "Push PROD ${{ inputs.cache-type }} cache: ${{ matrix.python-version }} ${{ inputs.platform }}"
- run: >
- breeze prod-image build --builder airflow_cache
- --prepare-buildx-cache --platform "${{ inputs.platform }}"
- --install-packages-from-context --airflow-constraints-mode constraints-source-providers
- --python ${{ matrix.python }}
- - name: "Stop ARM instance"
- run: ./scripts/ci/images/ci_stop_arm_instance.sh
- if: always() && inputs.platform == 'linux/arm64'
+ env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ ACTOR: ${{ github.actor }}
+ run: echo "${GITHUB_TOKEN}" | docker login ghcr.io -u ${ACTOR} --password-stdin
# We only push "AMD" images as it is really only needed for any kind of automated builds in CI
# and currently there is not an easy way to make multi-platform image from two separate builds
# and we can do it after we stopped the ARM instance as it is not needed anymore
- - name: "Push PROD latest image: ${{ matrix.python }} (linux/amd64 ONLY)"
+ - name: "Push PROD latest image: ${{ env.PYTHON_MAJOR_MINOR_VERSION }} (linux/amd64 ONLY)"
+ env:
+ PLATFORM: ${{ inputs.platform }}
run: >
- breeze prod-image build --builder airflow_cache --install-packages-from-context
- --push --platform "${{ inputs.platform }}"
+ breeze prod-image build
+ --builder airflow_cache
+ --install-packages-from-context
+ --platform "${PLATFORM}"
+ --airflow-constraints-mode constraints-source-providers
if: inputs.push-latest-images == 'true' && inputs.platform == 'linux/amd64'
+ # yamllint disable-line rule:line-length
+ - name: "Push PROD ${{ inputs.cache-type }} cache: ${{ env.PYTHON_MAJOR_MINOR_VERSION }} ${{ inputs.platform }}"
+ env:
+ PLATFORM: ${{ inputs.platform }}
+ run: >
+ breeze prod-image build
+ --builder airflow_cache
+ --prepare-buildx-cache
+ --install-packages-from-context
+ --platform "${PLATFORM}"
+ --airflow-constraints-mode constraints-source-providers
+ --push
diff --git a/.github/workflows/recheck-old-bug-report.yml b/.github/workflows/recheck-old-bug-report.yml
index ee14cfde5f7..217092b86f8 100644
--- a/.github/workflows/recheck-old-bug-report.yml
+++ b/.github/workflows/recheck-old-bug-report.yml
@@ -45,6 +45,7 @@ jobs:
remove-stale-when-updated: false
remove-issue-stale-when-updated: true
labels-to-add-when-unstale: 'needs-triage'
+ labels-to-remove-when-unstale: 'Stale Bug Report'
stale-issue-message: >
This issue has been automatically marked as stale because it has been open for 365 days
without any activity. There has been several Airflow releases since last activity on this issue.
diff --git a/.github/workflows/release_dockerhub_image.yml b/.github/workflows/release_dockerhub_image.yml
index 5ce1585131f..b8758146cc1 100644
--- a/.github/workflows/release_dockerhub_image.yml
+++ b/.github/workflows/release_dockerhub_image.yml
@@ -63,11 +63,14 @@ jobs:
run: ./scripts/ci/cleanup_docker.sh
- name: "Install Breeze"
uses: ./.github/actions/breeze
+ with:
+ use-uv: "false"
- name: Selective checks
id: selective-checks
env:
VERBOSE: "false"
run: breeze ci selective-check 2>> ${GITHUB_OUTPUT}
+
release-images:
timeout-minutes: 120
name: "Release images: ${{ github.event.inputs.airflowVersion }}, ${{ matrix.python-version }}"
@@ -99,6 +102,8 @@ jobs:
run: ./scripts/ci/cleanup_docker.sh
- name: "Install Breeze"
uses: ./.github/actions/breeze
+ with:
+ use-uv: "false"
- name: Free space
run: breeze ci free-space --answer yes
- name: "Cleanup dist and context file"
@@ -108,7 +113,10 @@ jobs:
echo ${{ secrets.DOCKERHUB_TOKEN }} |
docker login --password-stdin --username ${{ secrets.DOCKERHUB_USER }}
- name: Login to ghcr.io
- run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
+ env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ ACTOR: ${{ github.actor }}
+ run: echo "${GITHUB_TOKEN}" | docker login ghcr.io -u ${ACTOR} --password-stdin
- name: "Install buildx plugin"
# yamllint disable rule:line-length
run: |
@@ -141,10 +149,12 @@ jobs:
# from the source code, not from the PyPI because they have apache-airflow>=X.Y.Z dependency
# And when we prepare them from sources they will have apache-airflow>=X.Y.Z.dev0
shell: bash
+ env:
+ CHICKEN_EGG_PROVIDERS: ${{ needs.build-info.outputs.chicken-egg-providers }}
run: >
breeze release-management prepare-provider-packages
--package-format wheel
- --version-suffix-for-pypi dev0 ${{ needs.build-info.outputs.chicken-egg-providers }}
+ --version-suffix-for-pypi dev0 ${CHICKEN_EGG_PROVIDERS}
if: needs.build-info.outputs.chicken-egg-providers != ''
- name: "Copy dist packages to docker-context files"
shell: bash
@@ -152,42 +162,61 @@ jobs:
if: needs.build-info.outputs.chicken-egg-providers != ''
- name: >
Release regular images: ${{ github.event.inputs.airflowVersion }}, ${{ matrix.python-version }}
- run: >
- breeze release-management release-prod-images
- --dockerhub-repo ${{ github.repository }}
- --airflow-version ${{ github.event.inputs.airflowVersion }}
- ${{ needs.build-info.outputs.skipLatest }}
- ${{ needs.build-info.outputs.limitPlatform }}
- --limit-python ${{ matrix.python-version }}
- --chicken-egg-providers "${{ needs.build-info.outputs.chicken-egg-providers }}"
env:
COMMIT_SHA: ${{ github.sha }}
- - name: >
- Release slim images: ${{ github.event.inputs.airflowVersion }}, ${{ matrix.python-version }}
+ REPOSITORY: ${{ github.repository }}
+ PYTHON_VERSION: ${{ matrix.python-version }}
+ AIRFLOW_VERSION: ${{ github.event.inputs.airflowVersion }}
+ SKIP_LATEST: ${{ needs.build-info.outputs.skipLatest }}
+ LIMIT_PLATFORM: ${{ needs.build-info.outputs.limitPlatform }}
+ CHICKEN_EGG_PROVIDERS: ${{ needs.build-info.outputs.chicken-egg-providers }}
run: >
breeze release-management release-prod-images
- --dockerhub-repo ${{ github.repository }}
- --airflow-version ${{ github.event.inputs.airflowVersion }}
- ${{ needs.build-info.outputs.skipLatest }}
- ${{ needs.build-info.outputs.limitPlatform }}
- --limit-python ${{ matrix.python-version }} --slim-images
+ --dockerhub-repo "${REPOSITORY}"
+ --airflow-version "${AIRFLOW_VERSION}"
+ ${SKIP_LATEST}
+ ${LIMIT_PLATFORM}
+ --limit-python ${PYTHON_VERSION}
+ --chicken-egg-providers ${CHICKEN_EGG_PROVIDERS}
+ - name: >
+ Release slim images: ${{ github.event.inputs.airflowVersion }}, ${{ matrix.python-version }}
env:
COMMIT_SHA: ${{ github.sha }}
+ REPOSITORY: ${{ github.repository }}
+ PYTHON_VERSION: ${{ matrix.python-version }}
+ AIRFLOW_VERSION: ${{ github.event.inputs.airflowVersion }}
+ SKIP_LATEST: ${{ needs.build-info.outputs.skipLatest }}
+ LIMIT_PLATFORM: ${{ needs.build-info.outputs.limitPlatform }}
+ run: >
+ breeze release-management release-prod-images
+ --dockerhub-repo "${REPOSITORY}"
+ --airflow-version "${AIRFLOW_VERSION}"
+ ${SKIP_LATEST}
+ ${LIMIT_PLATFORM}
+ --limit-python ${PYTHON_VERSION} --slim-images
- name: >
Verify regular AMD64 image: ${{ github.event.inputs.airflowVersion }}, ${{ matrix.python-version }}
+ env:
+ PYTHON_VERSION: ${{ matrix.python-version }}
+ AIRFLOW_VERSION: ${{ github.event.inputs.airflowVersion }}
+ REPOSITORY: ${{ github.repository }}
run: >
breeze prod-image verify
--pull
--image-name
- ${{github.repository}}:${{github.event.inputs.airflowVersion}}-python${{matrix.python-version}}
+ ${REPOSITORY}:${AIRFLOW_VERSION}-python${PYTHON_VERSION}
- name: >
Verify slim AMD64 image: ${{ github.event.inputs.airflowVersion }}, ${{ matrix.python-version }}
+ env:
+ PYTHON_VERSION: ${{ matrix.python-version }}
+ AIRFLOW_VERSION: ${{ github.event.inputs.airflowVersion }}
+ REPOSITORY: ${{ github.repository }}
run: >
breeze prod-image verify
--pull
--slim-image
--image-name
- ${{github.repository}}:slim-${{github.event.inputs.airflowVersion}}-python${{matrix.python-version}}
+ ${REPOSITORY}:slim-${AIRFLOW_VERSION}-python${PYTHON_VERSION}
- name: "Docker logout"
run: docker logout
if: always()
diff --git a/.github/workflows/run-unit-tests.yml b/.github/workflows/run-unit-tests.yml
index 2989a952d9e..1c24e659d09 100644
--- a/.github/workflows/run-unit-tests.yml
+++ b/.github/workflows/run-unit-tests.yml
@@ -24,6 +24,10 @@ on: # yamllint disable-line rule:truthy
description: "The array of labels (in json form) determining default runner used for the build."
required: true
type: string
+ test-groups:
+ description: "The json representing list of test test groups to run"
+ required: true
+ type: string
backend:
description: "The backend to run the tests on"
required: true
@@ -41,10 +45,6 @@ on: # yamllint disable-line rule:truthy
required: false
default: ":"
type: string
- image-tag:
- description: "Tag to set for the image"
- required: true
- type: string
python-versions:
description: "The list of python versions (stringified JSON array) to run the tests on."
required: true
@@ -53,12 +53,20 @@ on: # yamllint disable-line rule:truthy
description: "The list of backend versions (stringified JSON array) to run the tests on."
required: true
type: string
+ excluded-providers-as-string:
+ description: "Excluded providers (per Python version) as json string"
+ required: true
+ type: string
excludes:
description: "Excluded combos (stringified JSON array of python-version/backend-version dicts)"
required: true
type: string
- parallel-test-types-list-as-string:
- description: "The list of parallel test types to run separated by spaces"
+ core-test-types-list-as-string:
+ description: "The list of core test types to run separated by spaces"
+ required: true
+ type: string
+ providers-test-types-list-as-string:
+ description: "The list of providers test types to run separated by spaces"
required: true
type: string
run-migration-tests:
@@ -89,26 +97,11 @@ on: # yamllint disable-line rule:truthy
required: false
default: "false"
type: string
- pydantic:
- description: "The version of pydantic to use"
- required: false
- default: "v2"
- type: string
downgrade-pendulum:
description: "Whether to downgrade pendulum or not (true/false)"
required: false
default: "false"
type: string
- enable-aip-44:
- description: "Whether to enable AIP-44 or not (true/false)"
- required: false
- default: "true"
- type: string
- database-isolation:
- description: "Whether to enable database isolattion or not (true/false)"
- required: false
- default: "false"
- type: string
force-lowest-dependencies:
description: "Whether to force lowest dependencies for the tests or not (true/false)"
required: false
@@ -119,13 +112,17 @@ on: # yamllint disable-line rule:truthy
required: false
default: 20
type: number
+ use-uv:
+ description: "Whether to use uv"
+ required: true
+ type: string
jobs:
tests:
timeout-minutes: 120
name: "\
- ${{ inputs.test-scope }}:\
+ ${{ inputs.test-scope }}-${{ matrix.test-group }}:\
${{ inputs.test-name }}${{ inputs.test-name-separator }}${{ matrix.backend-version }}:\
- ${{matrix.python-version}}: ${{ inputs.parallel-test-types-list-as-string }}"
+ ${{matrix.python-version}}"
runs-on: ${{ fromJSON(inputs.runs-on-as-json-default) }}
strategy:
fail-fast: false
@@ -133,9 +130,8 @@ jobs:
python-version: "${{fromJSON(inputs.python-versions)}}"
backend-version: "${{fromJSON(inputs.backend-versions)}}"
exclude: "${{fromJSON(inputs.excludes)}}"
+ test-group: "${{fromJSON(inputs.test-groups)}}"
env:
- # yamllint disable rule:line-length
- AIRFLOW_ENABLE_AIP_44: "${{ inputs.enable-aip-44 }}"
BACKEND: "${{ inputs.backend }}"
BACKEND_VERSION: "${{ matrix.backend-version }}"
DB_RESET: "true"
@@ -143,21 +139,20 @@ jobs:
DOWNGRADE_SQLALCHEMY: "${{ inputs.downgrade-sqlalchemy }}"
DOWNGRADE_PENDULUM: "${{ inputs.downgrade-pendulum }}"
ENABLE_COVERAGE: "${{ inputs.run-coverage }}"
+ EXCLUDED_PROVIDERS: "${{ inputs.excluded-providers-as-string }}"
FORCE_LOWEST_DEPENDENCIES: "${{ inputs.force-lowest-dependencies }}"
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_USERNAME: ${{ github.actor }}
- IMAGE_TAG: "${{ inputs.image-tag }}"
INCLUDE_SUCCESS_OUTPUTS: ${{ inputs.include-success-outputs }}
# yamllint disable rule:line-length
- JOB_ID: "${{ inputs.test-scope }}-${{ inputs.test-name }}-${{inputs.backend}}-${{ matrix.backend-version }}-${{ matrix.python-version }}"
+ JOB_ID: "${{ matrix.test-group }}-${{ inputs.test-scope }}-${{ inputs.test-name }}-${{inputs.backend}}-${{ matrix.backend-version }}-${{ matrix.python-version }}"
MOUNT_SOURCES: "skip"
- PARALLEL_TEST_TYPES: "${{ inputs.parallel-test-types-list-as-string }}"
- PYDANTIC: "${{ inputs.pydantic }}"
+ # yamllint disable rule:line-length
+ PARALLEL_TEST_TYPES: ${{ matrix.test-group == 'core' && inputs.core-test-types-list-as-string || inputs.providers-test-types-list-as-string }}
PYTHON_MAJOR_MINOR_VERSION: "${{ matrix.python-version }}"
UPGRADE_BOTO: "${{ inputs.upgrade-boto }}"
AIRFLOW_MONITOR_DELAY_TIME_IN_SECONDS: "${{inputs.monitor-delay-time-in-seconds}}"
- DATABASE_ISOLATION: "${{ inputs.database-isolation }}"
VERBOSE: "true"
steps:
- name: "Cleanup repo"
@@ -167,38 +162,23 @@ jobs:
uses: actions/checkout@v4
with:
persist-credentials: false
- - name: "Cleanup docker"
- run: ./scripts/ci/cleanup_docker.sh
- - name: "Prepare breeze & CI image: ${{matrix.python-version}}:${{ inputs.image-tag }}"
+ - name: "Prepare breeze & CI image: ${{ matrix.python-version }}"
uses: ./.github/actions/prepare_breeze_and_image
+ with:
+ platform: "linux/amd64"
+ python: ${{ matrix.python-version }}
+ use-uv: ${{ inputs.use-uv }}
- name: >
- Migration Tests:
- ${{ matrix.python-version }}:${{ inputs.parallel-test-types-list-as-string }}
+ Migration Tests: ${{ matrix.python-version }}:${{ env.PARALLEL_TEST_TYPES }}
uses: ./.github/actions/migration_tests
- if: inputs.run-migration-tests == 'true'
+ if: inputs.run-migration-tests == 'true' && matrix.test-group == 'core'
- name: >
- ${{ inputs.test-scope }} Tests ${{ inputs.test-name }} ${{ matrix.backend-version }}
- Py${{ matrix.python-version }}:${{ inputs.parallel-test-types-list-as-string}}
- run: |
- if [[ "${{ inputs.test-scope }}" == "DB" ]]; then
- breeze testing db-tests \
- --parallel-test-types "${{ inputs.parallel-test-types-list-as-string }}"
- elif [[ "${{ inputs.test-scope }}" == "Non-DB" ]]; then
- breeze testing non-db-tests \
- --parallel-test-types "${{ inputs.parallel-test-types-list-as-string }}"
- elif [[ "${{ inputs.test-scope }}" == "All" ]]; then
- breeze testing tests --run-in-parallel \
- --parallel-test-types "${{ inputs.parallel-test-types-list-as-string }}"
- elif [[ "${{ inputs.test-scope }}" == "Quarantined" ]]; then
- breeze testing tests --test-type "All-Quarantined" || true
- elif [[ "${{ inputs.test-scope }}" == "ARM collection" ]]; then
- breeze testing tests --collect-only --remove-arm-packages
- elif [[ "${{ inputs.test-scope }}" == "System" ]]; then
- breeze testing tests tests/system/example_empty.py --system core
- else
- echo "Unknown test scope: ${{ inputs.test-scope }}"
- exit 1
- fi
+ ${{ matrix.test-group}}:${{ inputs.test-scope }} Tests ${{ inputs.test-name }} ${{ matrix.backend-version }}
+ Py${{ matrix.python-version }}:${{ env.PARALLEL_TEST_TYPES }}
+ env:
+ TEST_GROUP: "${{ matrix.test-group }}"
+ TEST_SCOPE: "${{ inputs.test-scope }}"
+ run: ./scripts/ci/testing/run_unit_tests.sh "${TEST_GROUP}" "${TEST_SCOPE}"
- name: "Post Tests success"
uses: ./.github/actions/post_tests_success
with:
@@ -207,4 +187,4 @@ jobs:
if: success()
- name: "Post Tests failure"
uses: ./.github/actions/post_tests_failure
- if: failure()
+ if: failure() || cancelled()
diff --git a/.github/workflows/special-tests.yml b/.github/workflows/special-tests.yml
index 9a2a4cad330..36ccbf871cc 100644
--- a/.github/workflows/special-tests.yml
+++ b/.github/workflows/special-tests.yml
@@ -28,12 +28,16 @@ on: # yamllint disable-line rule:truthy
description: "The default branch for the repository"
required: true
type: string
- image-tag:
- description: "Tag to set for the image"
+ test-groups:
+ description: "The json representing list of test test groups to run"
required: true
type: string
- parallel-test-types-list-as-string:
- description: "The list of parallel test types to run separated by spaces"
+ core-test-types-list-as-string:
+ description: "The list of core test types to run separated by spaces"
+ required: true
+ type: string
+ providers-test-types-list-as-string:
+ description: "The list of providers test types to run separated by spaces"
required: true
type: string
run-coverage:
@@ -44,6 +48,10 @@ on: # yamllint disable-line rule:truthy
description: "Which version of python should be used by default"
required: true
type: string
+ excluded-providers-as-string:
+ description: "Excluded providers (per Python version) as json string"
+ required: true
+ type: string
python-versions:
description: "The list of python versions (stringified JSON array) to run the tests on."
required: true
@@ -60,10 +68,19 @@ on: # yamllint disable-line rule:truthy
description: "Whether to upgrade to newer dependencies or not (true/false)"
required: true
type: string
+ include-success-outputs:
+ description: "Whether to include success outputs or not (true/false)"
+ required: true
+ type: string
debug-resources:
description: "Whether to debug resources or not (true/false)"
required: true
type: string
+ use-uv:
+ description: "Whether to use uv or not (true/false)"
+ required: true
+ type: string
+
jobs:
tests-min-sqlalchemy:
name: "Min SQLAlchemy test"
@@ -77,15 +94,17 @@ jobs:
downgrade-sqlalchemy: "true"
test-name: "MinSQLAlchemy-Postgres"
test-scope: "DB"
+ test-groups: ${{ inputs.test-groups }}
backend: "postgres"
- image-tag: ${{ inputs.image-tag }}
python-versions: "['${{ inputs.default-python-version }}']"
backend-versions: "['${{ inputs.default-postgres-version }}']"
+ excluded-providers-as-string: ${{ inputs.excluded-providers-as-string }}
excludes: "[]"
- parallel-test-types-list-as-string: ${{ inputs.parallel-test-types-list-as-string }}
- include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }}
+ core-test-types-list-as-string: ${{ inputs.core-test-types-list-as-string }}
+ providers-test-types-list-as-string: ${{ inputs.providers-test-types-list-as-string }}
run-coverage: ${{ inputs.run-coverage }}
debug-resources: ${{ inputs.debug-resources }}
+ use-uv: ${{ inputs.use-uv }}
tests-boto:
name: "Latest Boto test"
@@ -99,59 +118,18 @@ jobs:
upgrade-boto: "true"
test-name: "LatestBoto-Postgres"
test-scope: "All"
+ test-groups: ${{ inputs.test-groups }}
backend: "postgres"
- image-tag: ${{ inputs.image-tag }}
python-versions: "['${{ inputs.default-python-version }}']"
backend-versions: "['${{ inputs.default-postgres-version }}']"
+ excluded-providers-as-string: ${{ inputs.excluded-providers-as-string }}
excludes: "[]"
- parallel-test-types-list-as-string: ${{ inputs.parallel-test-types-list-as-string }}
- include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }}
- run-coverage: ${{ inputs.run-coverage }}
- debug-resources: ${{ inputs.debug-resources }}
-
- tests-pydantic-v1:
- name: "Pydantic v1 test"
- uses: ./.github/workflows/run-unit-tests.yml
- permissions:
- contents: read
- packages: read
- secrets: inherit
- with:
- runs-on-as-json-default: ${{ inputs.runs-on-as-json-default }}
- pydantic: "v1"
- test-name: "Pydantic-V1-Postgres"
- test-scope: "All"
- backend: "postgres"
- image-tag: ${{ inputs.image-tag }}
- python-versions: "['${{ inputs.default-python-version }}']"
- backend-versions: "['${{ inputs.default-postgres-version }}']"
- excludes: "[]"
- parallel-test-types-list-as-string: ${{ inputs.parallel-test-types-list-as-string }}
- include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }}
- run-coverage: ${{ inputs.run-coverage }}
- debug-resources: ${{ inputs.debug-resources }}
-
- tests-pydantic-none:
- name: "Pydantic removed test"
- uses: ./.github/workflows/run-unit-tests.yml
- permissions:
- contents: read
- packages: read
- secrets: inherit
- with:
- runs-on-as-json-default: ${{ inputs.runs-on-as-json-default }}
- pydantic: "none"
- test-name: "Pydantic-Removed-Postgres"
- test-scope: "All"
- backend: "postgres"
- image-tag: ${{ inputs.image-tag }}
- python-versions: "['${{ inputs.default-python-version }}']"
- backend-versions: "['${{ inputs.default-postgres-version }}']"
- excludes: "[]"
- parallel-test-types-list-as-string: ${{ inputs.parallel-test-types-list-as-string }}
- include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }}
+ core-test-types-list-as-string: ${{ inputs.core-test-types-list-as-string }}
+ providers-test-types-list-as-string: ${{ inputs.providers-test-types-list-as-string }}
+ include-success-outputs: ${{ inputs.include-success-outputs }}
run-coverage: ${{ inputs.run-coverage }}
debug-resources: ${{ inputs.debug-resources }}
+ use-uv: ${{ inputs.use-uv }}
tests-pendulum-2:
name: "Pendulum2 test"
@@ -165,60 +143,18 @@ jobs:
downgrade-pendulum: "true"
test-name: "Pendulum2-Postgres"
test-scope: "All"
+ test-groups: ${{ inputs.test-groups }}
backend: "postgres"
- image-tag: ${{ inputs.image-tag }}
- python-versions: "['${{ inputs.default-python-version }}']"
- backend-versions: "['${{ inputs.default-postgres-version }}']"
- excludes: "[]"
- parallel-test-types-list-as-string: ${{ inputs.parallel-test-types-list-as-string }}
- include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }}
- run-coverage: ${{ inputs.run-coverage }}
- debug-resources: ${{ inputs.debug-resources }}
-
- tests-in-progress-disabled:
- name: "In progress disabled test"
- uses: ./.github/workflows/run-unit-tests.yml
- permissions:
- contents: read
- packages: read
- secrets: inherit
- with:
- runs-on-as-json-default: ${{ inputs.runs-on-as-json-default }}
- enable-aip-44: "false"
- test-name: "InProgressDisabled-Postgres"
- test-scope: "All"
- backend: "postgres"
- image-tag: ${{ inputs.image-tag }}
- python-versions: "['${{ inputs.default-python-version }}']"
- backend-versions: "['${{ inputs.default-postgres-version }}']"
- excludes: "[]"
- parallel-test-types-list-as-string: ${{ inputs.parallel-test-types-list-as-string }}
- include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }}
- run-coverage: ${{ inputs.run-coverage }}
- debug-resources: ${{ inputs.debug-resources }}
-
- tests-database-isolation:
- name: "Database isolation test"
- uses: ./.github/workflows/run-unit-tests.yml
- permissions:
- contents: read
- packages: read
- secrets: inherit
- with:
- runs-on-as-json-default: ${{ inputs.runs-on-as-json-default }}
- enable-aip-44: "true"
- database-isolation: "true"
- test-name: "DatabaseIsolation-Postgres"
- test-scope: "DB"
- backend: "postgres"
- image-tag: ${{ inputs.image-tag }}
python-versions: "['${{ inputs.default-python-version }}']"
backend-versions: "['${{ inputs.default-postgres-version }}']"
+ excluded-providers-as-string: ${{ inputs.excluded-providers-as-string }}
excludes: "[]"
- parallel-test-types-list-as-string: ${{ inputs.parallel-test-types-list-as-string }}
- include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }}
+ core-test-types-list-as-string: ${{ inputs.core-test-types-list-as-string }}
+ providers-test-types-list-as-string: ${{ inputs.providers-test-types-list-as-string }}
+ include-success-outputs: ${{ inputs.include-success-outputs }}
run-coverage: ${{ inputs.run-coverage }}
debug-resources: ${{ inputs.debug-resources }}
+ use-uv: ${{ inputs.use-uv }}
tests-quarantined:
name: "Quarantined test"
@@ -231,15 +167,18 @@ jobs:
runs-on-as-json-default: ${{ inputs.runs-on-as-json-default }}
test-name: "Postgres"
test-scope: "Quarantined"
+ test-groups: ${{ inputs.test-groups }}
backend: "postgres"
- image-tag: ${{ inputs.image-tag }}
python-versions: "['${{ inputs.default-python-version }}']"
backend-versions: "['${{ inputs.default-postgres-version }}']"
+ excluded-providers-as-string: ${{ inputs.excluded-providers-as-string }}
excludes: "[]"
- parallel-test-types-list-as-string: ${{ inputs.parallel-test-types-list-as-string }}
- include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }}
+ core-test-types-list-as-string: ${{ inputs.core-test-types-list-as-string }}
+ providers-test-types-list-as-string: ${{ inputs.providers-test-types-list-as-string }}
+ include-success-outputs: ${{ inputs.include-success-outputs }}
run-coverage: ${{ inputs.run-coverage }}
debug-resources: ${{ inputs.debug-resources }}
+ use-uv: ${{ inputs.use-uv }}
tests-arm-collection:
name: "ARM Collection test"
@@ -252,19 +191,23 @@ jobs:
runs-on-as-json-default: ${{ inputs.runs-on-as-json-default }}
test-name: "Postgres"
test-scope: "ARM collection"
+ test-groups: ${{ inputs.test-groups }}
backend: "postgres"
- image-tag: ${{ inputs.image-tag }}
python-versions: "['${{ inputs.default-python-version }}']"
backend-versions: "['${{ inputs.default-postgres-version }}']"
+ excluded-providers-as-string: ${{ inputs.excluded-providers-as-string }}
excludes: "[]"
- parallel-test-types-list-as-string: ${{ inputs.parallel-test-types-list-as-string }}
- include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }}
+ core-test-types-list-as-string: ${{ inputs.core-test-types-list-as-string }}
+ providers-test-types-list-as-string: ${{ inputs.providers-test-types-list-as-string }}
+ include-success-outputs: ${{ inputs.include-success-outputs }}
run-coverage: ${{ inputs.run-coverage }}
debug-resources: ${{ inputs.debug-resources }}
+ use-uv: ${{ inputs.use-uv }}
if: ${{ inputs.default-branch == 'main' }}
+ # matrix.test-group comes from run-unit-tests.yml
tests-system:
- name: "System test"
+ name: "System test: ${{ matrix.test-group }}"
uses: ./.github/workflows/run-unit-tests.yml
permissions:
contents: read
@@ -274,12 +217,15 @@ jobs:
runs-on-as-json-default: ${{ inputs.runs-on-as-json-default }}
test-name: "SystemTest"
test-scope: "System"
+ test-groups: ${{ inputs.test-groups }}
backend: "postgres"
- image-tag: ${{ inputs.image-tag }}
python-versions: "['${{ inputs.default-python-version }}']"
backend-versions: "['${{ inputs.default-postgres-version }}']"
+ excluded-providers-as-string: ${{ inputs.excluded-providers-as-string }}
excludes: "[]"
- parallel-test-types-list-as-string: ${{ inputs.parallel-test-types-list-as-string }}
- include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }}
+ core-test-types-list-as-string: ${{ inputs.core-test-types-list-as-string }}
+ providers-test-types-list-as-string: ${{ inputs.providers-test-types-list-as-string }}
+ include-success-outputs: ${{ inputs.include-success-outputs }}
run-coverage: ${{ inputs.run-coverage }}
debug-resources: ${{ inputs.debug-resources }}
+ use-uv: ${{ inputs.use-uv }}
diff --git a/.github/workflows/task-sdk-tests.yml b/.github/workflows/task-sdk-tests.yml
new file mode 100644
index 00000000000..501e880fd3b
--- /dev/null
+++ b/.github/workflows/task-sdk-tests.yml
@@ -0,0 +1,93 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: Task SDK tests
+on: # yamllint disable-line rule:truthy
+ workflow_call:
+ inputs:
+ runs-on-as-json-default:
+ description: "The array of labels (in json form) determining default runner used for the build."
+ required: true
+ type: string
+ default-python-version:
+ description: "Which version of python should be used by default"
+ required: true
+ type: string
+ python-versions:
+ description: "JSON-formatted array of Python versions to build images from"
+ required: true
+ type: string
+ run-task-sdk-tests:
+ description: "Whether to run Task SDK tests or not (true/false)"
+ required: true
+ type: string
+ use-uv:
+ description: "Whether to use uv to build the image (true/false)"
+ required: true
+ type: string
+ canary-run:
+ description: "Whether this is a canary run (true/false)"
+ required: true
+ type: string
+
+jobs:
+ task-sdk-tests:
+ timeout-minutes: 80
+ name: Task SDK:P${{ matrix.python-version }} tests
+ runs-on: ${{ fromJSON(inputs.runs-on-as-json-default) }}
+ strategy:
+ fail-fast: false
+ matrix:
+ python-version: "${{fromJSON(inputs.python-versions)}}"
+ env:
+ GITHUB_REPOSITORY: ${{ github.repository }}
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ GITHUB_USERNAME: ${{ github.actor }}
+ INCLUDE_NOT_READY_PROVIDERS: "true"
+ PYTHON_MAJOR_MINOR_VERSION: "${{ inputs.default-python-version }}"
+ VERBOSE: "true"
+ CLEAN_AIRFLOW_INSTALLATION: "${{ inputs.canary-run }}"
+ steps:
+ - name: "Cleanup repo"
+ shell: bash
+ run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ uses: actions/checkout@v4
+ with:
+ persist-credentials: false
+ - name: "Prepare breeze & CI image: ${{ matrix.python-version }}"
+ uses: ./.github/actions/prepare_breeze_and_image
+ with:
+ platform: "linux/amd64"
+ python: ${{ matrix.python-version }}
+ use-uv: ${{ inputs.use-uv }}
+ - name: "Cleanup dist files"
+ run: rm -fv ./dist/*
+ - name: "Prepare Task SDK packages: wheel"
+ run: >
+ breeze release-management prepare-task-sdk-package --package-format wheel
+ - name: "Verify wheel packages with twine"
+ run: |
+ uv tool uninstall twine || true
+ uv tool install twine && twine check dist/*.whl
+ - name: >
+ Run unit tests for Airflow Task SDK:Python ${{ matrix.python-version }}
+ env:
+ PYTHON_VERSION: "${{ matrix.python-version }}"
+ run: >
+ breeze testing task-sdk-tests --python "${PYTHON_VERSION}"
diff --git a/.github/workflows/check-providers.yml b/.github/workflows/test-provider-packages.yml
similarity index 57%
rename from .github/workflows/check-providers.yml
rename to .github/workflows/test-provider-packages.yml
index e89d4a81faa..877ff1f1b23 100644
--- a/.github/workflows/check-providers.yml
+++ b/.github/workflows/test-provider-packages.yml
@@ -24,10 +24,6 @@ on: # yamllint disable-line rule:truthy
description: "The array of labels (in json form) determining default runner used for the build."
required: true
type: string
- image-tag:
- description: "Tag to set for the image"
- required: true
- type: string
canary-run:
description: "Whether this is a canary run"
required: true
@@ -40,13 +36,13 @@ on: # yamllint disable-line rule:truthy
description: "Whether to upgrade to newer dependencies"
required: true
type: string
- affected-providers-list-as-string:
+ selected-providers-list-as-string:
description: "List of affected providers as string"
required: false
type: string
- providers-compatibility-checks:
+ providers-compatibility-tests-matrix:
description: >
- JSON-formatted array of providers compatibility checks in the form of array of dicts
+ JSON-formatted array of providers compatibility tests in the form of array of dicts
(airflow-version, python-versions, remove-providers, run-tests)
required: true
type: string
@@ -54,7 +50,7 @@ on: # yamllint disable-line rule:truthy
description: "List of parallel provider test types as string"
required: true
type: string
- skip-provider-tests:
+ skip-providers-tests:
description: "Whether to skip provider tests (true/false)"
required: true
type: string
@@ -62,16 +58,23 @@ on: # yamllint disable-line rule:truthy
description: "JSON-formatted array of Python versions to build images from"
required: true
type: string
+ use-uv:
+ description: "Whether to use uv"
+ required: true
+ type: string
jobs:
- prepare-install-verify-provider-packages-wheel:
+ prepare-install-verify-provider-packages:
timeout-minutes: 80
- name: "Provider packages wheel build and verify"
+ name: "Providers ${{ matrix.package-format }} tests"
runs-on: ${{ fromJSON(inputs.runs-on-as-json-default) }}
+ strategy:
+ fail-fast: false
+ matrix:
+ package-format: ["wheel", "sdist"]
env:
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_USERNAME: ${{ github.actor }}
- IMAGE_TAG: "${{ inputs.image-tag }}"
INCLUDE_NOT_READY_PROVIDERS: "true"
PYTHON_MAJOR_MINOR_VERSION: "${{ inputs.default-python-version }}"
VERBOSE: "true"
@@ -83,138 +86,93 @@ jobs:
uses: actions/checkout@v4
with:
persist-credentials: false
- - name: "Cleanup docker"
- run: ./scripts/ci/cleanup_docker.sh
- - name: >
- Prepare breeze & CI image: ${{ inputs.default-python-version }}:${{ inputs.image-tag }}
+ - name: "Prepare breeze & CI image: ${{ inputs.default-python-version }}"
uses: ./.github/actions/prepare_breeze_and_image
+ with:
+ platform: "linux/amd64"
+ python: ${{ inputs.default-python-version }}
+ use-uv: ${{ inputs.use-uv }}
- name: "Cleanup dist files"
run: rm -fv ./dist/*
- name: "Prepare provider documentation"
run: >
breeze release-management prepare-provider-documentation --include-not-ready-providers
--non-interactive
- - name: "Prepare provider packages: wheel"
+ if: matrix.package-format == 'wheel'
+ - name: "Prepare provider packages: ${{ matrix.package-format }}"
run: >
breeze release-management prepare-provider-packages --include-not-ready-providers
- --version-suffix-for-pypi dev0 --package-format wheel
- - name: "Prepare airflow package: wheel"
- run: breeze release-management prepare-airflow-package --version-suffix-for-pypi dev0
- - name: "Verify wheel packages with twine"
+ --version-suffix-for-pypi dev0 --package-format ${{ matrix.package-format }}
+ - name: "Prepare airflow package: ${{ matrix.package-format }}"
+ run: >
+ breeze release-management prepare-airflow-package --version-suffix-for-pypi dev0
+ --package-format ${{ matrix.package-format }}
+ - name: "Verify ${{ matrix.package-format }} packages with twine"
run: |
- pipx uninstall twine || true
- pipx install twine && twine check dist/*.whl
+ uv tool uninstall twine || true
+ uv tool install twine && twine check dist/*
- name: "Test providers issue generation automatically"
run: >
breeze release-management generate-issue-content-providers
--only-available-in-dist --disable-progress
+ if: matrix.package-format == 'wheel'
+ - name: Remove Python 3.9-incompatible provider packages
+ run: |
+ echo "Removing Python 3.9-incompatible provider: cloudant"
+ rm -vf dist/*cloudant*
- name: "Generate source constraints from CI image"
shell: bash
run: >
breeze release-management generate-constraints
--airflow-constraints-mode constraints-source-providers --answer yes
- - name: "Install and verify all provider packages and airflow via wheel files"
- run: >
- breeze release-management verify-provider-packages
- --use-packages-from-dist
- --package-format wheel
- --use-airflow-version wheel
- --airflow-constraints-reference default
- --providers-constraints-location
- /files/constraints-${{env.PYTHON_MAJOR_MINOR_VERSION}}/constraints-source-providers-${{env.PYTHON_MAJOR_MINOR_VERSION}}.txt
+ - name: "Install and verify wheel provider packages"
env:
+ PACKAGE_FORMAT: ${{ matrix.package-format }}
+ PYTHON_MAJOR_MINOR_VERSION: ${env.PYTHON_MAJOR_MINOR_VERSION}
AIRFLOW_SKIP_CONSTRAINTS: "${{ inputs.upgrade-to-newer-dependencies }}"
- - name: "Prepare airflow package: wheel without suffix and skipping the tag check"
run: >
- breeze release-management prepare-provider-packages --skip-tag-check --package-format wheel
-
- prepare-install-provider-packages-sdist:
- timeout-minutes: 80
- name: "Provider packages sdist build and install"
- runs-on: ${{ fromJSON(inputs.runs-on-as-json-default) }}
- env:
- GITHUB_REPOSITORY: ${{ github.repository }}
- GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- GITHUB_USERNAME: ${{ github.actor }}
- IMAGE_TAG: "${{ inputs.image-tag }}"
- INCLUDE_NOT_READY_PROVIDERS: "true"
- PYTHON_MAJOR_MINOR_VERSION: "${{ inputs.default-python-version }}"
- VERBOSE: "true"
- steps:
- - name: "Cleanup repo"
- shell: bash
- run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
- uses: actions/checkout@v4
- with:
- persist-credentials: false
- - name: "Cleanup docker"
- run: ./scripts/ci/cleanup_docker.sh
- - name: >
- Prepare breeze & CI image: ${{ inputs.default-python-version }}:${{ inputs.image-tag }}
- uses: ./.github/actions/prepare_breeze_and_image
- - name: "Cleanup dist files"
- run: rm -fv ./dist/*
- - name: "Prepare provider packages: sdist"
- run: >
- breeze release-management prepare-provider-packages --include-not-ready-providers
- --version-suffix-for-pypi dev0 --package-format sdist
- ${{ inputs.affected-providers-list-as-string }}
- - name: "Prepare airflow package: sdist"
- run: >
- breeze release-management prepare-airflow-package
- --version-suffix-for-pypi dev0 --package-format sdist
- - name: "Verify sdist packages with twine"
- run: |
- pipx uninstall twine || true
- pipx install twine && twine check dist/*.tar.gz
- - name: "Generate source constraints from CI image"
- shell: bash
- run: >
- breeze release-management generate-constraints
- --airflow-constraints-mode constraints-source-providers --answer yes
- - name: "Install all provider packages and airflow via sdist files"
- run: >
- breeze release-management install-provider-packages
+ breeze release-management verify-provider-packages
--use-packages-from-dist
- --package-format sdist
- --use-airflow-version sdist
+ --package-format "${PACKAGE_FORMAT}"
+ --use-airflow-version "${PACKAGE_FORMAT}"
--airflow-constraints-reference default
--providers-constraints-location
- /files/constraints-${{env.PYTHON_MAJOR_MINOR_VERSION}}/constraints-source-providers-${{env.PYTHON_MAJOR_MINOR_VERSION}}.txt
- --run-in-parallel
- if: inputs.affected-providers-list-as-string == ''
- - name: "Install affected provider packages and airflow via sdist files"
+ /files/constraints-${PYTHON_MAJOR_MINOR_VERSION}/constraints-source-providers-${PYTHON_MAJOR_MINOR_VERSION}.txt
+ if: matrix.package-format == 'wheel'
+ - name: "Install all sdist provider packages and airflow"
+ env:
+ PACKAGE_FORMAT: ${{ matrix.package-format }}
+ PYTHON_MAJOR_MINOR_VERSION: ${{ env.PYTHON_MAJOR_MINOR_VERSION }}
run: >
breeze release-management install-provider-packages
--use-packages-from-dist
- --package-format sdist
- --use-airflow-version sdist
+ --package-format "${PACKAGE_FORMAT}"
+ --use-airflow-version ${PACKAGE_FORMAT}
--airflow-constraints-reference default
--providers-constraints-location
- /files/constraints-${{env.PYTHON_MAJOR_MINOR_VERSION}}/constraints-source-providers-${{env.PYTHON_MAJOR_MINOR_VERSION}}.txt
+ /files/constraints-${PYTHON_MAJOR_MINOR_VERSION}/constraints-source-providers-${PYTHON_MAJOR_MINOR_VERSION}.txt
--run-in-parallel
- if: inputs.affected-providers-list-as-string != ''
+ if: matrix.package-format == 'sdist'
- providers-compatibility-checks:
+ # All matrix parameters are passed as JSON string in the input variable providers-compatibility-tests-matrix
+ providers-compatibility-tests-matrix:
timeout-minutes: 80
- name: Compat ${{ matrix.airflow-version }}:P${{ matrix.python-version }} provider check
+ name: Compat ${{ matrix.airflow-version }}:P${{ matrix.python-version }} providers test
runs-on: ${{ fromJSON(inputs.runs-on-as-json-default) }}
strategy:
fail-fast: false
matrix:
- include: ${{fromJSON(inputs.providers-compatibility-checks)}}
+ include: ${{fromJSON(inputs.providers-compatibility-tests-matrix)}}
env:
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_USERNAME: ${{ github.actor }}
- IMAGE_TAG: "${{ inputs.image-tag }}"
INCLUDE_NOT_READY_PROVIDERS: "true"
- PYTHON_MAJOR_MINOR_VERSION: "${{ inputs.default-python-version }}"
+ PYTHON_MAJOR_MINOR_VERSION: "${{ matrix.python-version }}"
VERSION_SUFFIX_FOR_PYPI: "dev0"
VERBOSE: "true"
CLEAN_AIRFLOW_INSTALLATION: "${{ inputs.canary-run }}"
- if: inputs.skip-provider-tests != 'true'
+ if: inputs.skip-providers-tests != 'true'
steps:
- name: "Cleanup repo"
shell: bash
@@ -223,10 +181,12 @@ jobs:
uses: actions/checkout@v4
with:
persist-credentials: false
- - name: "Cleanup docker"
- run: ./scripts/ci/cleanup_docker.sh
- - name: "Prepare breeze & CI image: ${{ matrix.python-version }}:${{ inputs.image-tag }}"
+ - name: "Prepare breeze & CI image: ${{ matrix.python-version }}"
uses: ./.github/actions/prepare_breeze_and_image
+ with:
+ platform: "linux/amd64"
+ python: ${{ matrix.python-version }}
+ use-uv: ${{ inputs.use-uv }}
- name: "Cleanup dist files"
run: rm -fv ./dist/*
- name: "Prepare provider packages: wheel"
@@ -236,8 +196,10 @@ jobs:
- name: >
Remove incompatible Airflow
${{ matrix.airflow-version }}:Python ${{ matrix.python-version }} provider packages
+ env:
+ REMOVE_PROVIDERS: ${{ matrix.remove-providers }}
run: |
- for provider in ${{ matrix.remove-providers }}; do
+ for provider in ${REMOVE_PROVIDERS}; do
echo "Removing incompatible provider: ${provider}"
rm -vf dist/apache_airflow_providers_${provider/./_}*
done
@@ -251,25 +213,34 @@ jobs:
# We do not need to run import check if we run tests, the tests should cover all the import checks
# automatically
if: matrix.run-tests != 'true'
+ env:
+ AIRFLOW_VERSION: "${{ matrix.airflow-version }}"
run: >
breeze release-management verify-provider-packages
--use-packages-from-dist
--package-format wheel
--use-airflow-version wheel
- --airflow-constraints-reference constraints-${{matrix.airflow-version}}
+ --airflow-constraints-reference constraints-${AIRFLOW_VERSION}
--providers-skip-constraints
--install-airflow-with-constraints
+ - name: Check amount of disk space available
+ run: df -H
+ shell: bash
- name: >
Run provider unit tests on
Airflow ${{ matrix.airflow-version }}:Python ${{ matrix.python-version }}
if: matrix.run-tests == 'true'
+ env:
+ PROVIDERS_TEST_TYPES: "${{ inputs.providers-test-types-list-as-string }}"
+ AIRFLOW_VERSION: "${{ matrix.airflow-version }}"
+ REMOVE_PROVIDERS: "${{ matrix.remove-providers }}"
run: >
- breeze testing tests --run-in-parallel
- --parallel-test-types "${{ inputs.providers-test-types-list-as-string }}"
+ breeze testing providers-tests --run-in-parallel
+ --parallel-test-types "${PROVIDERS_TEST_TYPES}"
--use-packages-from-dist
--package-format wheel
- --use-airflow-version "${{ matrix.airflow-version }}"
- --airflow-constraints-reference constraints-${{matrix.airflow-version}}
+ --use-airflow-version "${AIRFLOW_VERSION}"
+ --airflow-constraints-reference constraints-${AIRFLOW_VERSION}
--install-airflow-with-constraints
--providers-skip-constraints
- --skip-providers "${{ matrix.remove-providers }}"
+ --skip-providers "${REMOVE_PROVIDERS}"
diff --git a/.gitignore b/.gitignore
index ac6ea156611..84afbd47410 100644
--- a/.gitignore
+++ b/.gitignore
@@ -12,6 +12,7 @@ airflow.db
airflow/git_version
airflow/www/static/coverage/
airflow/www/*.log
+airflow/ui/coverage/
logs/
airflow-webserver.pid
standalone_admin_password.txt
@@ -110,6 +111,7 @@ celerybeat-schedule
# dotenv
.env
+.env.local
.autoenv*.zsh
# virtualenv
@@ -127,9 +129,6 @@ ENV/
.idea/
*.iml
-# Visual Studio Code
-.vscode/
-
# vim
*.swp
@@ -165,6 +164,15 @@ node_modules
npm-debug.log*
derby.log
metastore_db
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+pnpm-debug.log*
+.vscode/*
+!.vscode/extensions.json
+/.vite/
+.pnpm-store
+*.tsbuildinfo
# Airflow log files when airflow is run locally
airflow-*.err
@@ -249,6 +257,3 @@ airflow-build-dockerfile*
# Temporary ignore uv.lock until we integrate it fully in our constraint preparation mechanism
/uv.lock
-
-# Airflow 3 related files
-airflow/ui
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 027fbd006b9..f1b6a3fe714 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -20,11 +20,13 @@ default_language_version:
python: python3
node: 22.2.0
minimum_pre_commit_version: '3.2.0'
+exclude: ^.*/.*_vendor/
repos:
- repo: meta
hooks:
- id: identity
- name: Print input to the static check hooks for troubleshooting
+ name: Print checked files
+ description: Print input to the static check hooks for troubleshooting
- id: check-hooks-apply
name: Check if all hooks apply to the repository
- repo: https://github.com/thlorenz/doctoc.git
@@ -34,7 +36,6 @@ repos:
name: Add TOC for Markdown and RST files
files:
^README\.md$|^UPDATING.*\.md$|^chart/UPDATING.*\.md$|^dev/.*\.md$|^dev/.*\.rst$|^.github/.*\.md|^tests/system/README.md$
- exclude: ^.*/.*_vendor/
args:
- "--maxlevel"
- "2"
@@ -46,8 +47,7 @@ repos:
files: \.sql$
exclude: |
(?x)
- ^\.github/|
- ^.*/.*_vendor/
+ ^\.github/
args:
- --comment-style
- "/*||*/"
@@ -56,7 +56,7 @@ repos:
- --fuzzy-match-generates-todo
- id: insert-license
name: Add license for all RST files
- exclude: ^\.github/.*$|^.*/.*_vendor/|newsfragments/.*\.rst$
+ exclude: ^\.github/.*$|newsfragments/.*\.rst$
args:
- --comment-style
- "||"
@@ -65,9 +65,9 @@ repos:
- --fuzzy-match-generates-todo
files: \.rst$
- id: insert-license
- name: Add license for all CSS/JS/JSX/PUML/TS/TSX files
+ name: Add license for CSS/JS/JSX/PUML/TS/TSX
files: \.(css|jsx?|puml|tsx?)$
- exclude: ^\.github/.*$|^.*/.*_vendor/
+ exclude: ^\.github/.*$|^airflow/www/static/js/types/api-generated.ts$|ui/openapi-gen/
args:
- --comment-style
- "/*!| *| */"
@@ -77,7 +77,7 @@ repos:
- id: insert-license
name: Add license for all JINJA template files
files: ^airflow/www/templates/.*\.html$
- exclude: ^\.github/.*$|^.*/.*_vendor/
+ exclude: ^\.github/.*$
args:
- --comment-style
- "{#||#}"
@@ -86,7 +86,7 @@ repos:
- --fuzzy-match-generates-todo
- id: insert-license
name: Add license for all Shell files
- exclude: ^\.github/.*$|^.*/.*_vendor/|^dev/breeze/autocomplete/.*$
+ exclude: ^\.github/.*$|^dev/breeze/autocomplete/.*$
files: \.bash$|\.sh$
args:
- --comment-style
@@ -96,7 +96,7 @@ repos:
- --fuzzy-match-generates-todo
- id: insert-license
name: Add license for all toml files
- exclude: ^\.github/.*$|^.*/.*_vendor/|^dev/breeze/autocomplete/.*$
+ exclude: ^\.github/.*$|^dev/breeze/autocomplete/.*$
files: \.toml$
args:
- --comment-style
@@ -106,7 +106,7 @@ repos:
- --fuzzy-match-generates-todo
- id: insert-license
name: Add license for all Python files
- exclude: ^\.github/.*$|^.*/.*_vendor/
+ exclude: ^\.github/.*$
files: \.py$|\.pyi$
args:
- --comment-style
@@ -116,7 +116,7 @@ repos:
- --fuzzy-match-generates-todo
- id: insert-license
name: Add license for all XML files
- exclude: ^\.github/.*$|^.*/.*_vendor/
+ exclude: ^\.github/.*$
files: \.xml$
args:
- --comment-style
@@ -135,7 +135,7 @@ repos:
- --fuzzy-match-generates-todo
- id: insert-license
name: Add license for all YAML files except Helm templates
- exclude: ^\.github/.*$|^.*/.*_vendor/|^chart/templates/.*|.*/reproducible_build.yaml$
+ exclude: ^\.github/.*$|^chart/templates/.*|.*/reproducible_build.yaml$|^airflow/api_fastapi/core_api/openapi/v1-generated.yaml$|^.*/pnpm-lock.yaml$
types: [yaml]
files: \.ya?ml$
args:
@@ -147,7 +147,7 @@ repos:
- id: insert-license
name: Add license for all Markdown files
files: \.md$
- exclude: PROVIDER_CHANGES.*\.md$|^.*/.*_vendor/
+ exclude: PROVIDER_CHANGES.*\.md$
args:
- --comment-style
- "<!--|| -->"
@@ -156,7 +156,7 @@ repos:
- --fuzzy-match-generates-todo
- id: insert-license
name: Add license for all other files
- exclude: ^\.github/.*$|^.*/.*_vendor/
+ exclude: ^\.github/.*$
args:
- --comment-style
- "|#|"
@@ -173,14 +173,6 @@ repos:
language: python
additional_dependencies: ['rich>=12.4.4']
require_serial: true
- - id: update-common-sql-api-stubs
- name: Check and update common.sql API stubs
- entry: ./scripts/ci/pre_commit/update_common_sql_api_stubs.py
- language: python
- files: ^scripts/ci/pre_commit/update_common_sql_api\.py|^airflow/providers/common/sql/.*\.pyi?$
- additional_dependencies: ['rich>=12.4.4', 'mypy==1.9.0', 'black==23.10.0', 'jinja2']
- pass_filenames: false
- require_serial: true
- id: update-black-version
name: Update black versions everywhere (manual)
entry: ./scripts/ci/pre_commit/update_black_version.py
@@ -190,21 +182,12 @@ repos:
additional_dependencies: ['pyyaml']
pass_filenames: false
require_serial: true
- - id: update-build-dependencies
- name: Update build-dependencies to latest (manual)
- entry: ./scripts/ci/pre_commit/update_build_dependencies.py
- stages: ['manual']
- language: python
- files: ^.pre-commit-config.yaml$|^scripts/ci/pre_commit/update_build_dependencies.py$
- pass_filenames: false
- require_serial: true
- additional_dependencies: ['rich>=12.4.4']
- - id: update-installers
- name: Update installers to latest (manual)
- entry: ./scripts/ci/pre_commit/update_installers.py
+ - id: update-installers-and-pre-commit
+ name: Update installers and pre-commit to latest (manual)
+ entry: ./scripts/ci/pre_commit/update_installers_and_pre_commit.py
stages: ['manual']
language: python
- files: ^.pre-commit-config.yaml$|^scripts/ci/pre_commit/update_installers.py$
+ files: ^.pre-commit-config.yaml$|^scripts/ci/pre_commit/update_installers_and_pre_commit.py$
pass_filenames: false
require_serial: true
additional_dependencies: ['pyyaml', 'rich>=12.4.4', 'requests']
@@ -233,49 +216,44 @@ repos:
additional_dependencies: ["libcst>=1.1.0"]
files: ^airflow/.*/sensors/.*\.py$|^airflow/.*/operators/.*\.py$
- repo: https://github.com/asottile/blacken-docs
- rev: 1.18.0
+ rev: 1.19.1
hooks:
- id: blacken-docs
- name: Run black on Python code blocks in documentation files
+ name: Run black on docs
args:
- --line-length=110
- - --target-version=py37
- - --target-version=py38
- --target-version=py39
- --target-version=py310
+ - --target-version=py311
+ - --target-version=py312
alias: blacken-docs
- additional_dependencies: [black==23.10.0]
+ additional_dependencies: [black==24.10.0]
- repo: https://github.com/pre-commit/pre-commit-hooks
- rev: v4.6.0
+ rev: v5.0.0
hooks:
- id: check-merge-conflict
name: Check that merge conflicts are not being committed
- id: debug-statements
name: Detect accidentally committed debug statements
- id: check-builtin-literals
- name: Require literal syntax when initializing builtin types
- exclude: ^.*/.*_vendor/
+ name: Require literal syntax when initializing builtins
- id: detect-private-key
name: Detect if private key is added to the repository
exclude: ^docs/apache-airflow-providers-ssh/connections/ssh.rst$
- id: end-of-file-fixer
name: Make sure that there is an empty line at the end
- exclude: ^.*/.*_vendor/|^docs/apache-airflow/img/.*\.dot|^docs/apache-airflow/img/.*\.sha256
+ exclude: ^docs/apache-airflow/img/.*\.dot|^docs/apache-airflow/img/.*\.sha256
- id: mixed-line-ending
name: Detect if mixed line ending is used (\r vs. \r\n)
- exclude: ^.*/.*_vendor/
- id: check-executables-have-shebangs
name: Check that executables have shebang
- exclude: ^.*/.*_vendor/
- id: check-xml
name: Check XML files with xmllint
- exclude: ^.*/.*_vendor/
- id: trailing-whitespace
name: Remove trailing whitespace at end of line
- exclude: ^.*/.*_vendor/|^docs/apache-airflow/img/.*\.dot|^dev/breeze/doc/images/output.*$
+ exclude: ^docs/apache-airflow/img/.*\.dot|^dev/breeze/doc/images/output.*$
- id: fix-encoding-pragma
name: Remove encoding header from Python files
- exclude: ^.*/.*_vendor/
args:
- --remove
- id: pretty-format-json
@@ -292,10 +270,8 @@ repos:
hooks:
- id: rst-backticks
name: Check if RST files use double backticks for code
- exclude: ^.*/.*_vendor/
- id: python-no-log-warn
name: Check if there are no deprecate log warn
- exclude: ^.*/.*_vendor/
- repo: https://github.com/adrienverge/yamllint
rev: v1.35.1
hooks:
@@ -303,15 +279,12 @@ repos:
name: Check YAML files with yamllint
entry: yamllint -c yamllint-config.yml --strict
types: [yaml]
- exclude: ^.*airflow\.template\.yaml$|^.*init_git_sync\.template\.yaml$|^.*/.*_vendor/|^chart/(?:templates|files)/.*\.yaml$|openapi/.*\.yaml$|^\.pre-commit-config\.yaml$|^.*/reproducible_build.yaml$
+ exclude: ^.*airflow\.template\.yaml$|^.*init_git_sync\.template\.yaml$|^chart/(?:templates|files)/.*\.yaml$|openapi/.*\.yaml$|^\.pre-commit-config\.yaml$|^.*/reproducible_build.yaml$|^.*pnpm-lock\.yaml$
- repo: https://github.com/ikamensh/flynt
rev: '1.0.1'
hooks:
- id: flynt
name: Run flynt string format converter for Python
- exclude: |
- (?x)
- ^.*/.*_vendor/
args:
# If flynt detects too long text it ignores it. So we set a very large limit to make it easy
# to split the text by hand. Too long lines are detected by flake8 (below),
@@ -322,17 +295,27 @@ repos:
rev: v2.3.0
hooks:
- id: codespell
- name: Run codespell to check for common misspellings in files
+ name: Run codespell
+ description: Run codespell to check for common misspellings in files
entry: bash -c 'echo "If you think that this failure is an error, consider adding the word(s)
to the codespell dictionary at docs/spelling_wordlist.txt.
The word(s) should be in lowercase." && exec codespell "$@"' --
language: python
types: [text]
- exclude: ^.*/.*_vendor/|^airflow/www/static/css/material-icons\.css$|^images/.*$|^RELEASE_NOTES\.txt$|^.*package-lock\.json$|^.*/kinglear\.txt$
+ exclude: material-icons\.css$|^images/.*$|^RELEASE_NOTES\.txt$|^.*package-lock\.json$|^.*/kinglear\.txt$|^.*pnpm-lock\.yaml$
args:
- --ignore-words=docs/spelling_wordlist.txt
- --skip=airflow/providers/*/*.rst,airflow/www/*.log,docs/*/commits.rst,docs/apache-airflow/tutorial/pipeline_example.csv,*.min.js,*.lock,INTHEWILD.md
- --exclude-file=.codespellignorelines
+ - repo: https://github.com/woodruffw/zizmor-pre-commit
+ rev: v1.0.0
+ hooks:
+ - id: zizmor
+ name: Run zizmor to check for github workflow syntax errors
+ types: [yaml]
+ files: \.github/workflows/.*$|\.github/actions/.*$
+ require_serial: true
+ entry: zizmor
- repo: local
# Note that this is the 2nd "local" repo group in the .pre-commit-config.yaml file. This is because
# we try to minimise the number of passes that must happen in order to apply some of the changes
@@ -342,13 +325,6 @@ repos:
# changes quickly - especially when we want the early modifications from the first local group
# to be applied before the non-local pre-commits are run
hooks:
- - id: validate-operators-init
- name: Prevent templated field logic checks in operators' __init__
- language: python
- entry: ./scripts/ci/pre_commit/validate_operators_init.py
- pass_filenames: true
- files: ^airflow/providers/.*/(operators|transfers|sensors)/.*\.py$
- additional_dependencies: [ 'rich>=12.4.4' ]
- id: ruff
name: Run 'ruff' for extremely fast Python linting
description: "Run 'ruff' for extremely fast Python linting"
@@ -357,24 +333,24 @@ repos:
types_or: [python, pyi]
args: [--fix]
require_serial: true
- additional_dependencies: ["ruff==0.5.5"]
- exclude: ^.*/.*_vendor/|^tests/dags/test_imports.py
+ additional_dependencies: ["ruff==0.8.1"]
+ exclude: ^tests/dags/test_imports.py|^performance/tests/test_.*.py
- id: ruff-format
- name: Run 'ruff format' for extremely fast Python formatting
+ name: Run 'ruff format'
description: "Run 'ruff format' for extremely fast Python formatting"
entry: ./scripts/ci/pre_commit/ruff_format.py
language: python
types_or: [python, pyi]
args: []
require_serial: true
- additional_dependencies: ["ruff==0.5.5"]
- exclude: ^.*/.*_vendor/|^tests/dags/test_imports.py|^airflow/contrib/
+ additional_dependencies: ["ruff==0.8.1"]
+ exclude: ^tests/dags/test_imports.py$
- id: replace-bad-characters
name: Replace bad characters
entry: ./scripts/ci/pre_commit/replace_bad_characters.py
language: python
types: [file, text]
- exclude: ^.*/.*_vendor/|^clients/gen/go\.sh$|^\.gitmodules$
+ exclude: ^clients/gen/go\.sh$|^\.gitmodules$
additional_dependencies: ['rich>=12.4.4']
- id: lint-openapi
name: Lint OpenAPI using spectral
@@ -410,36 +386,6 @@ repos:
exclude: ^airflow/openlineage/
entry: ./scripts/ci/pre_commit/check_common_compat_used_for_openlineage.py
additional_dependencies: ['rich>=12.4.4']
- - id: check-airflow-providers-bug-report-template
- name: Check airflow-bug-report provider list is sorted/unique
- language: python
- files: ^.github/ISSUE_TEMPLATE/airflow_providers_bug_report\.yml$
- require_serial: true
- entry: ./scripts/ci/pre_commit/check_airflow_bug_report_template.py
- additional_dependencies: ['rich>=12.4.4', 'pyyaml']
- - id: check-cncf-k8s-only-for-executors
- name: Check cncf.kubernetes imports used for executors only
- language: python
- files: ^airflow/.*\.py$
- require_serial: true
- exclude: ^airflow/kubernetes/|^airflow/providers/
- entry: ./scripts/ci/pre_commit/check_cncf_k8s_used_for_k8s_executor_only.py
- additional_dependencies: ['rich>=12.4.4']
- - id: check-airflow-provider-compatibility
- name: Check compatibility of Providers with Airflow
- entry: ./scripts/ci/pre_commit/check_provider_airflow_compatibility.py
- language: python
- pass_filenames: true
- files: ^airflow/providers/.*\.py$
- additional_dependencies: ['rich>=12.4.4']
- - id: check-google-re2-as-dependency
- name: Check google-re2 is declared as dependency when needed
- entry: ./scripts/ci/pre_commit/check_google_re2_imports.py
- language: python
- pass_filenames: true
- require_serial: true
- files: ^airflow/providers/.*\.py$
- additional_dependencies: ['rich>=12.4.4']
- id: update-local-yml-file
name: Update mounts in the local yml file
entry: ./scripts/ci/pre_commit/local_yml_mounts.py
@@ -447,12 +393,6 @@ repos:
files: ^dev/breeze/src/airflow_breeze/utils/docker_command_utils\.py$|^scripts/ci/docker_compose/local\.yml$
pass_filenames: false
additional_dependencies: ['rich>=12.4.4']
- - id: check-sql-dependency-common-data-structure
- name: Check dependency of SQL Providers with common data structure
- entry: ./scripts/ci/pre_commit/check_common_sql_dependency.py
- language: python
- files: ^airflow/providers/.*/hooks/.*\.py$
- additional_dependencies: ['rich>=12.4.4', 'pyyaml', 'packaging']
- id: update-providers-dependencies
name: Update dependencies for provider packages
entry: ./scripts/ci/pre_commit/update_providers_dependencies.py
@@ -475,13 +415,6 @@ repos:
pass_filenames: false
entry: ./scripts/ci/pre_commit/check_order_hatch_build.py
additional_dependencies: ['rich>=12.4.4', 'hatchling==1.27.0']
- - id: update-extras
- name: Update extras in documentation
- entry: ./scripts/ci/pre_commit/insert_extras.py
- language: python
- files: ^contributing-docs/12_airflow_dependencies_and_extras.rst$|^INSTALL$|^airflow/providers/.*/provider\.yaml$|^Dockerfile.*
- pass_filenames: false
- additional_dependencies: ['rich>=12.4.4', 'hatchling==1.27.0']
- id: check-extras-order
name: Check order of extras in Dockerfile
entry: ./scripts/ci/pre_commit/check_order_dockerfile_extras.py
@@ -503,19 +436,8 @@ repos:
files: ^docs/apache-airflow/installation/supported-versions\.rst$|^scripts/ci/pre_commit/supported_versions\.py$|^README\.md$
pass_filenames: false
additional_dependencies: ['tabulate']
- - id: check-revision-heads-map
- name: Check that the REVISION_HEADS_MAP is up-to-date
- language: python
- entry: ./scripts/ci/pre_commit/version_heads_map.py
- pass_filenames: false
- files: >
- (?x)
- ^scripts/ci/pre_commit/version_heads_map\.py$|
- ^airflow/migrations/versions/.*$|^airflow/migrations/versions|
- ^airflow/utils/db.py$
- additional_dependencies: ['packaging','google-re2']
- id: update-version
- name: Update version to the latest version in the documentation
+ name: Update versions in docs
entry: ./scripts/ci/pre_commit/update_versions.py
language: python
files: ^docs|^airflow/__init__.py$
@@ -528,7 +450,7 @@ repos:
pass_filenames: true
files: \.py$
- id: check-links-to-example-dags-do-not-use-hardcoded-versions
- name: Verify example dags do not use hard-coded version numbers
+ name: Verify no hard-coded version in example dags
description: The links to example dags should use |version| as version specification
language: pygrep
entry: >
@@ -622,13 +544,13 @@ repos:
^airflow/providers/opsgenie/hooks/opsgenie.py$|
^airflow/providers/redis/provider.yaml$|
^airflow/serialization/serialized_objects.py$|
+ ^airflow/ui/pnpm-lock.yaml$|
^airflow/utils/db.py$|
^airflow/utils/trigger_rule.py$|
^airflow/www/static/css/bootstrap-theme.css$|
^airflow/www/static/js/types/api-generated.ts$|
^airflow/www/templates/appbuilder/flash.html$|
^chart/values.schema.json$|
- ^.*/.*_vendor/|
^dev/|
^docs/README.rst$|
^docs/apache-airflow-providers-amazon/secrets-backends/aws-ssm-parameter-store.rst$|
@@ -641,14 +563,19 @@ repos:
^docs/apache-airflow-providers-cncf-kubernetes/operators.rst$|
^docs/conf.py$|
^docs/exts/removemarktransform.py$|
+ ^newsfragments/41761.significant.rst$|
^scripts/ci/pre_commit/vendor_k8s_json_schema.py$|
+ ^scripts/ci/docker-compose/integration-keycloak.yml$|
+ ^scripts/ci/docker-compose/keycloak/keycloak-entrypoint.sh$|
^tests/|
+ ^providers/tests/|
^.pre-commit-config\.yaml$|
^.*CHANGELOG\.(rst|txt)$|
^.*RELEASE_NOTES\.rst$|
^contributing-docs/03_contributors_quick_start.rst$|
^.*\.(png|gif|jp[e]?g|tgz|lock)$|
- git
+ git|
+ ^newsfragments/43349\.significant\.rst$
- id: check-base-operator-partial-arguments
name: Check BaseOperator and partial() arguments
language: python
@@ -656,13 +583,13 @@ repos:
pass_filenames: false
files: ^airflow/models/(?:base|mapped)operator\.py$
- id: check-init-decorator-arguments
- name: Check model __init__ and decorator arguments are in sync
+ name: Sync model __init__ and decorator arguments
language: python
entry: ./scripts/ci/pre_commit/sync_init_decorator.py
pass_filenames: false
files: ^airflow/models/dag\.py$|^airflow/(?:decorators|utils)/task_group\.py$
- id: check-template-context-variable-in-sync
- name: Check all template context variable references are in sync
+ name: Sync template context variable refs
language: python
entry: ./scripts/ci/pre_commit/template_context_key_sync.py
files: ^airflow/models/taskinstance\.py$|^airflow/utils/context\.pyi?$|^docs/apache-airflow/templates-ref\.rst$
@@ -730,33 +657,28 @@ repos:
pass_filenames: true
- id: check-provide-create-sessions-imports
language: pygrep
- name: Check provide_session and create_session imports
- description: provide_session and create_session should be imported from airflow.utils.session
- to avoid import cycles.
- entry: "from airflow\\.utils\\.db import.* (provide_session|create_session)"
+ name: Check session util imports
+ description: NEW_SESSION, provide_session, and create_session should be imported from airflow.utils.session to avoid import cycles.
+ entry: "from airflow\\.utils\\.db import.* (NEW_SESSION|provide_session|create_session)"
files: \.py$
- exclude: ^.*/.*_vendor/
pass_filenames: true
- id: check-incorrect-use-of-LoggingMixin
language: pygrep
name: Make sure LoggingMixin is not used alone
entry: "LoggingMixin\\(\\)"
files: \.py$
- exclude: ^.*/.*_vendor/
pass_filenames: true
- id: check-daysago-import-from-utils
language: pygrep
- name: Make sure days_ago is imported from airflow.utils.dates
+ name: days_ago imported from airflow.utils.dates
entry: "(airflow\\.){0,1}utils\\.dates\\.days_ago"
files: \.py$
- exclude: ^.*/.*_vendor/
pass_filenames: true
- id: check-start-date-not-used-in-defaults
language: pygrep
- name: start_date not to be defined in default_args in example_dags
+ name: start_date not in default_args
entry: "default_args\\s*=\\s*{\\s*(\"|')start_date(\"|')|(\"|')start_date(\"|'):"
files: \.*example_dags.*\.py$
- exclude: ^.*/.*_vendor/
pass_filenames: true
- id: check-apache-license-rat
name: Check if licenses are OK for Apache
@@ -778,7 +700,7 @@ repos:
entry: ./scripts/ci/pre_commit/boring_cyborg.py
pass_filenames: false
require_serial: true
- additional_dependencies: ['pyyaml', 'termcolor==1.1.0', 'wcmatch==8.2']
+ additional_dependencies: ['pyyaml', 'termcolor==2.5.0', 'wcmatch==8.2']
- id: update-in-the-wild-to-be-sorted
name: Sort INTHEWILD.md alphabetically
entry: ./scripts/ci/pre_commit/sort_in_the_wild.py
@@ -787,14 +709,14 @@ repos:
pass_filenames: false
require_serial: true
- id: update-installed-providers-to-be-sorted
- name: Sort alphabetically and uniquify installed_providers.txt
+ name: Sort and uniquify installed_providers.txt
entry: ./scripts/ci/pre_commit/sort_installed_providers.py
language: python
files: ^\.pre-commit-config\.yaml$|^.*_installed_providers\.txt$
pass_filenames: false
require_serial: true
- id: update-spelling-wordlist-to-be-sorted
- name: Sort alphabetically and uniquify spelling_wordlist.txt
+ name: Sort spelling_wordlist.txt
entry: ./scripts/ci/pre_commit/sort_spelling_wordlist.py
language: python
files: ^\.pre-commit-config\.yaml$|^docs/spelling_wordlist\.txt$
@@ -855,12 +777,6 @@ repos:
entry: ./scripts/ci/pre_commit/compile_www_assets_dev.py
pass_filenames: false
additional_dependencies: ['yarn@1.22.21']
- - id: check-providers-init-file-missing
- name: Provider init file is missing
- pass_filenames: false
- always_run: true
- entry: ./scripts/ci/pre_commit/check_providers_init.py
- language: python
- id: check-providers-subpackages-init-file-exist
name: Provider subpackage init files are there
pass_filenames: false
@@ -872,10 +788,10 @@ repos:
name: Validate hook IDs & names and sync with docs
entry: ./scripts/ci/pre_commit/check_pre_commit_hooks.py
args:
- - --max-length=60
+ - --max-length=53
language: python
files: ^\.pre-commit-config\.yaml$|^scripts/ci/pre_commit/check_pre_commit_hooks\.py$
- additional_dependencies: ['pyyaml', 'jinja2', 'black==23.10.0', 'tabulate', 'rich>=12.4.4']
+ additional_dependencies: ['pyyaml', 'jinja2', 'black==24.10.0', 'tabulate', 'rich>=12.4.4']
require_serial: true
pass_filenames: false
- id: check-integrations-list-consistent
@@ -883,7 +799,7 @@ repos:
entry: ./scripts/ci/pre_commit/check_integrations_list.py
language: python
files: ^scripts/ci/docker-compose/integration-.*\.yml$|^contributing-docs/testing/integration_tests.rst$
- additional_dependencies: ['black==23.10.0', 'tabulate', 'rich>=12.4.4', 'pyyaml']
+ additional_dependencies: ['black==24.10.0', 'tabulate', 'rich>=12.4.4', 'pyyaml']
require_serial: true
pass_filenames: false
- id: update-breeze-readme-config-hash
@@ -902,7 +818,8 @@ repos:
pass_filenames: false
require_serial: true
- id: check-breeze-top-dependencies-limited
- name: Breeze should have small number of top-level dependencies
+ name: Check top-level breeze deps
+ description: Breeze should have small number of top-level dependencies
language: python
entry: ./scripts/tools/check_if_limited_dependencies.py
files: ^dev/breeze/.*$
@@ -940,7 +857,7 @@ repos:
files: \.(md|mdown|markdown)$
additional_dependencies: ['markdownlint-cli@0.38.0']
- id: lint-json-schema
- name: Lint JSON Schema files with JSON Schema
+ name: Lint JSON Schema files
entry: ./scripts/ci/pre_commit/json_schema.py
args:
- --spec-file
@@ -948,11 +865,10 @@ repos:
language: python
pass_filenames: true
files: .*\.schema\.json$
- exclude: ^.*/.*_vendor/
require_serial: true
- additional_dependencies: ['jsonschema>=3.2.0,<5.0', 'PyYAML==5.3.1', 'requests==2.25.0']
+ additional_dependencies: ['jsonschema>=3.2.0,<5.0', 'PyYAML==6.0.2', 'requests==2.32.3']
- id: lint-json-schema
- name: Lint NodePort Service with JSON Schema
+ name: Lint NodePort Service
entry: ./scripts/ci/pre_commit/json_schema.py
args:
- --spec-url
@@ -961,9 +877,9 @@ repos:
pass_filenames: true
files: ^scripts/ci/kubernetes/nodeport\.yaml$
require_serial: true
- additional_dependencies: ['jsonschema>=3.2.0,<5.0', 'PyYAML==5.3.1', 'requests==2.25.0']
+ additional_dependencies: ['jsonschema>=3.2.0,<5.0', 'PyYAML==6.0.2', 'requests==2.32.3']
- id: lint-json-schema
- name: Lint Docker compose files with JSON Schema
+ name: Lint Docker compose files
entry: ./scripts/ci/pre_commit/json_schema.py
args:
- --spec-url
@@ -976,9 +892,9 @@ repos:
^scripts/ci/docker-compose/grafana/.|
^scripts/ci/docker-compose/.+-config\.ya?ml
require_serial: true
- additional_dependencies: ['jsonschema>=3.2.0,<5.0', 'PyYAML==5.3.1', 'requests==2.25.0']
+ additional_dependencies: ['jsonschema>=3.2.0,<5.0', 'PyYAML==6.0.2', 'requests==2.32.3']
- id: lint-json-schema
- name: Lint chart/values.schema.json file with JSON Schema
+ name: Lint chart/values.schema.json
entry: ./scripts/ci/pre_commit/json_schema.py
args:
- --spec-file
@@ -988,15 +904,15 @@ repos:
pass_filenames: false
files: ^chart/values\.schema\.json$|^chart/values_schema\.schema\.json$
require_serial: true
- additional_dependencies: ['jsonschema>=3.2.0,<5.0', 'PyYAML==5.3.1', 'requests==2.25.0']
+ additional_dependencies: ['jsonschema>=3.2.0,<5.0', 'PyYAML==6.0.2', 'requests==2.32.3']
- id: update-vendored-in-k8s-json-schema
name: Vendor k8s definitions into values.schema.json
entry: ./scripts/ci/pre_commit/vendor_k8s_json_schema.py
language: python
files: ^chart/values\.schema\.json$
- additional_dependencies: ['requests==2.25.0']
+ additional_dependencies: ['requests==2.32.3']
- id: lint-json-schema
- name: Lint chart/values.yaml file with JSON Schema
+ name: Lint chart/values.yaml
entry: ./scripts/ci/pre_commit/json_schema.py
args:
- --enforce-defaults
@@ -1007,9 +923,9 @@ repos:
pass_filenames: false
files: ^chart/values\.yaml$|^chart/values\.schema\.json$
require_serial: true
- additional_dependencies: ['jsonschema>=3.2.0,<5.0', 'PyYAML==5.3.1', 'requests==2.25.0']
+ additional_dependencies: ['jsonschema>=3.2.0,<5.0', 'PyYAML==6.0.2', 'requests==2.32.3']
- id: lint-json-schema
- name: Lint config_templates/config.yml file with JSON Schema
+ name: Lint config_templates/config.yml
entry: ./scripts/ci/pre_commit/json_schema.py
args:
- --spec-file
@@ -1018,9 +934,10 @@ repos:
pass_filenames: true
files: ^airflow/config_templates/config\.yml$
require_serial: true
- additional_dependencies: ['jsonschema>=3.2.0,<5.0', 'PyYAML==5.3.1', 'requests==2.25.0']
+ additional_dependencies: ['jsonschema>=3.2.0,<5.0', 'PyYAML==6.0.2', 'requests==2.32.3']
- id: check-persist-credentials-disabled-in-github-workflows
- name: Check that workflow files have persist-credentials disabled
+ name: Check persistent creds in workflow files
+ description: Check that workflow files have persist-credentials disabled
entry: ./scripts/ci/pre_commit/checkout_no_credentials.py
language: python
pass_filenames: true
@@ -1032,22 +949,14 @@ repos:
language: python
pass_filenames: true
files: \.py$
- exclude: ^.*/.*_vendor/
additional_dependencies: ['rich>=12.4.4']
- - id: check-compat-cache-on-methods
- name: Check that compat cache do not use on class methods
- entry: ./scripts/ci/pre_commit/compat_cache_on_methods.py
- language: python
- pass_filenames: true
- files: ^airflow/.*\.py$
- exclude: ^.*/.*_vendor/
- id: check-code-deprecations
name: Check deprecations categories in decorators
entry: ./scripts/ci/pre_commit/check_deprecations.py
language: python
pass_filenames: true
files: ^airflow/.*\.py$
- exclude: ^.*/.*_vendor/
+ additional_dependencies: ["rich>=12.4.4", "python-dateutil"]
- id: lint-chart-schema
name: Lint chart/values.schema.json file
entry: ./scripts/ci/pre_commit/chart_schema.py
@@ -1084,7 +993,8 @@ repos:
# This is fast, so not too much downside
always_run: true
- id: update-breeze-cmd-output
- name: Update output of breeze commands in Breeze documentation
+ name: Update breeze docs
+ description: Update output of breeze commands in Breeze documentation
entry: ./scripts/ci/pre_commit/breeze_cmd_line.py
language: python
files: >
@@ -1117,7 +1027,6 @@ repos:
language: python
pass_filenames: true
files: \.py$
- exclude: ^.*/.*_vendor/
additional_dependencies: ['rich>=12.4.4', 'astor']
- id: create-missing-init-py-files-tests
name: Create missing init.py files in tests
@@ -1127,15 +1036,17 @@ repos:
pass_filenames: false
files: ^tests/.*\.py$
- id: ts-compile-format-lint-www
- name: TS types generation / ESLint / Prettier against UI files
+ name: Compile / format / lint WWW
+ description: TS types generation / ESLint / Prettier against UI files
language: node
'types_or': [javascript, ts, tsx, yaml, css, json]
files: ^airflow/www/static/(js|css)/|^airflow/api_connexion/openapi/v1\.yaml$
- entry: ./scripts/ci/pre_commit/www_lint.py
+ entry: ./scripts/ci/pre_commit/lint_www.py
additional_dependencies: ['yarn@1.22.21', "openapi-typescript@>=6.7.4"]
pass_filenames: false
- id: check-tests-unittest-testcase
- name: Check that unit tests do not inherit from unittest.TestCase
+ name: Unit tests do not inherit from unittest.TestCase
+ description: Check that unit tests do not inherit from unittest.TestCase
entry: ./scripts/ci/pre_commit/unittest_testcase.py
language: python
pass_filenames: true
@@ -1224,39 +1135,23 @@ repos:
^airflow/migrations/versions/0124.*\.py$
## ADD MOST PRE-COMMITS ABOVE THAT LINE
# The below pre-commits are those requiring CI image to be built
- - id: mypy-dev
- name: Run mypy for dev
- language: python
- entry: ./scripts/ci/pre_commit/mypy.py
- files: ^dev/.*\.py$|^scripts/.*\.py$
- require_serial: true
- additional_dependencies: ['rich>=12.4.4']
- - id: mypy-dev
- stages: [ 'manual' ]
- name: Run mypy for dev (manual)
- language: python
- entry: ./scripts/ci/pre_commit/mypy_folder.py dev
- pass_filenames: false
- files: ^.*\.py$
- require_serial: true
- additional_dependencies: [ 'rich>=12.4.4' ]
- id: mypy-airflow
name: Run mypy for airflow
language: python
- entry: ./scripts/ci/pre_commit/mypy.py --namespace-packages
+ entry: ./scripts/ci/pre_commit/mypy.py
files: \.py$
exclude: ^.*/.*_vendor/|^airflow/migrations|^airflow/providers|^dev|^scripts|^docs|^provider_packages|^tests/providers|^tests/system/providers|^tests/dags/test_imports.py|^clients/python/test_.*\.py
require_serial: true
additional_dependencies: ['rich>=12.4.4']
- id: mypy-airflow
- stages: [ 'manual' ]
+ stages: ['manual']
name: Run mypy for airflow (manual)
language: python
entry: ./scripts/ci/pre_commit/mypy_folder.py airflow
pass_filenames: false
files: ^.*\.py$
require_serial: true
- additional_dependencies: [ 'rich>=12.4.4' ]
+ additional_dependencies: ['rich>=12.4.4']
- id: mypy-providers
name: Run mypy for providers
language: python
@@ -1269,7 +1164,7 @@ repos:
stages: ['manual']
name: Run mypy for providers (manual)
language: python
- entry: ./scripts/ci/pre_commit/mypy_folder.py airflow/providers
+ entry: ./scripts/ci/pre_commit/mypy_folder.py providers/src/airflow/providers
pass_filenames: false
files: ^.*\.py$
require_serial: true
diff --git a/Dockerfile b/Dockerfile
index 5ac5fd61e12..bf3bd7fd275 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -36,7 +36,7 @@
# much smaller.
#
# Use the same builder frontend version for everyone
-ARG AIRFLOW_EXTRAS="aiobotocore,amazon,async,celery,cncf-kubernetes,common-io,docker,elasticsearch,fab,ftp,google,google-auth,graphviz,grpc,hashicorp,http,ldap,microsoft-azure,mysql,odbc,openlineage,pandas,postgres,redis,sendgrid,sftp,slack,snowflake,ssh,statsd,uv,virtualenv"
+ARG AIRFLOW_EXTRAS="aiobotocore,amazon,async,celery,cncf-kubernetes,common-io,docker,elasticsearch,fab,ftp,google,google-auth,graphviz,grpc,hashicorp,http,ldap,microsoft-azure,mysql,odbc,openlineage,pandas,postgres,redis,sendgrid,sftp,slack,snowflake,ssh,statsd,uv"
ARG ADDITIONAL_AIRFLOW_EXTRAS=""
ARG ADDITIONAL_PYTHON_DEPS=""
@@ -45,12 +45,17 @@ ARG AIRFLOW_UID="50000"
ARG AIRFLOW_USER_HOME_DIR=/home/airflow
# latest released version here
-ARG AIRFLOW_VERSION="2.9.3"
+ARG AIRFLOW_VERSION="2.10.4"
ARG PYTHON_BASE_IMAGE="python:3.8-slim-bookworm"
+
+# You can swap comments between those two args to test pip from the main version
+# When you attempt to test if the version of `pip` from specified branch works for our builds
+# Also use `force pip` label on your PR to swap all places we use `uv` to `pip`
ARG AIRFLOW_PIP_VERSION=24.3.1
-ARG AIRFLOW_UV_VERSION=0.5.11
+# ARG AIRFLOW_PIP_VERSION="git+https://github.com/pypa/pip.git@main"
+ARG AIRFLOW_UV_VERSION=0.5.14
ARG AIRFLOW_USE_UV="false"
ARG UV_HTTP_TIMEOUT="300"
ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow"
@@ -417,85 +422,6 @@ common::show_packaging_tool_version_and_location
common::install_packaging_tools
EOF
-# The content below is automatically copied from scripts/docker/install_airflow_dependencies_from_branch_tip.sh
-COPY <<"EOF" /install_airflow_dependencies_from_branch_tip.sh
-#!/usr/bin/env bash
-
-. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
-
-: "${AIRFLOW_REPO:?Should be set}"
-: "${AIRFLOW_BRANCH:?Should be set}"
-: "${INSTALL_MYSQL_CLIENT:?Should be true or false}"
-: "${INSTALL_POSTGRES_CLIENT:?Should be true or false}"
-
-function install_airflow_dependencies_from_branch_tip() {
- echo
- echo "${COLOR_BLUE}Installing airflow from ${AIRFLOW_BRANCH}. It is used to cache dependencies${COLOR_RESET}"
- echo
- if [[ ${INSTALL_MYSQL_CLIENT} != "true" ]]; then
- AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS/mysql,}
- fi
- if [[ ${INSTALL_POSTGRES_CLIENT} != "true" ]]; then
- AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS/postgres,}
- fi
- local TEMP_AIRFLOW_DIR
- TEMP_AIRFLOW_DIR=$(mktemp -d)
- # Install latest set of dependencies - without constraints. This is to download a "base" set of
- # dependencies that we can cache and reuse when installing airflow using constraints and latest
- # pyproject.toml in the next step (when we install regular airflow).
- set -x
- curl -fsSL "https://github.com/${AIRFLOW_REPO}/archive/${AIRFLOW_BRANCH}.tar.gz" | \
- tar xz -C "${TEMP_AIRFLOW_DIR}" --strip 1
- # Make sure editable dependencies are calculated when devel-ci dependencies are installed
- ${PACKAGING_TOOL_CMD} install ${EXTRA_INSTALL_FLAGS} ${ADDITIONAL_PIP_INSTALL_FLAGS} \
- --editable "${TEMP_AIRFLOW_DIR}[${AIRFLOW_EXTRAS}]"
- set +x
- common::install_packaging_tools
- set -x
- echo "${COLOR_BLUE}Uninstalling providers. Dependencies remain${COLOR_RESET}"
- # Uninstall airflow and providers to keep only the dependencies. In the future when
- # planned https://github.com/pypa/pip/issues/11440 is implemented in pip we might be able to use this
- # flag and skip the remove step.
- pip freeze | grep apache-airflow-providers | xargs ${PACKAGING_TOOL_CMD} uninstall ${EXTRA_UNINSTALL_FLAGS} || true
- set +x
- echo
- echo "${COLOR_BLUE}Uninstalling just airflow. Dependencies remain. Now target airflow can be reinstalled using mostly cached dependencies${COLOR_RESET}"
- echo
- set +x
- ${PACKAGING_TOOL_CMD} uninstall ${EXTRA_UNINSTALL_FLAGS} apache-airflow
- rm -rf "${TEMP_AIRFLOW_DIR}"
- set -x
- # If you want to make sure dependency is removed from cache in your PR when you removed it from
- # pyproject.toml - please add your dependency here as a list of strings
- # for example:
- # DEPENDENCIES_TO_REMOVE=("package_a" "package_b")
- # Once your PR is merged, you should make a follow-up PR to remove it from this list
- # and increase the AIRFLOW_CI_BUILD_EPOCH in Dockerfile.ci to make sure your cache is rebuilt.
- local DEPENDENCIES_TO_REMOVE
- # IMPORTANT!! Make sure to increase AIRFLOW_CI_BUILD_EPOCH in Dockerfile.ci when you remove a dependency from that list
- DEPENDENCIES_TO_REMOVE=()
- if [[ "${DEPENDENCIES_TO_REMOVE[*]}" != "" ]]; then
- echo
- echo "${COLOR_BLUE}Uninstalling just removed dependencies (temporary until cache refreshes)${COLOR_RESET}"
- echo "${COLOR_BLUE}Dependencies to uninstall: ${DEPENDENCIES_TO_REMOVE[*]}${COLOR_RESET}"
- echo
- set +x
- ${PACKAGING_TOOL_CMD} uninstall "${DEPENDENCIES_TO_REMOVE[@]}" || true
- set -x
- # make sure that the dependency is not needed by something else
- pip check
- fi
-}
-
-common::get_colors
-common::get_packaging_tool
-common::get_airflow_version_specification
-common::get_constraints_location
-common::show_packaging_tool_version_and_location
-
-install_airflow_dependencies_from_branch_tip
-EOF
-
# The content below is automatically copied from scripts/docker/common.sh
COPY <<"EOF" /common.sh
#!/usr/bin/env bash
@@ -1374,7 +1300,8 @@ ARG PYTHON_BASE_IMAGE
ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE} \
DEBIAN_FRONTEND=noninteractive LANGUAGE=C.UTF-8 LANG=C.UTF-8 LC_ALL=C.UTF-8 \
LC_CTYPE=C.UTF-8 LC_MESSAGES=C.UTF-8 \
- PIP_CACHE_DIR=/tmp/.cache/pip
+ PIP_CACHE_DIR=/tmp/.cache/pip \
+ UV_CACHE_DIR=/tmp/.cache/uv
ARG DEV_APT_DEPS=""
ARG ADDITIONAL_DEV_APT_DEPS=""
@@ -1440,9 +1367,6 @@ ARG DEFAULT_CONSTRAINTS_BRANCH="constraints-main"
# By default PIP has progress bar but you can disable it.
ARG PIP_PROGRESS_BAR
-# By default we do not use pre-cached packages, but in CI/Breeze environment we override this to speed up
-# builds in case pyproject.toml changed. This is pure optimisation of CI/Breeze builds.
-ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="false"
# This is airflow version that is put in the label of the image build
ARG AIRFLOW_VERSION
# By default latest released version of airflow is installed (when empty) but this value can be overridden
@@ -1480,7 +1404,6 @@ ENV AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} \
AIRFLOW_UV_VERSION=${AIRFLOW_UV_VERSION} \
UV_HTTP_TIMEOUT=${UV_HTTP_TIMEOUT} \
AIRFLOW_USE_UV=${AIRFLOW_USE_UV} \
- AIRFLOW_PRE_CACHED_PIP_PACKAGES=${AIRFLOW_PRE_CACHED_PIP_PACKAGES} \
AIRFLOW_VERSION=${AIRFLOW_VERSION} \
AIRFLOW_INSTALLATION_METHOD=${AIRFLOW_INSTALLATION_METHOD} \
AIRFLOW_VERSION_SPECIFICATION=${AIRFLOW_VERSION_SPECIFICATION} \
@@ -1505,8 +1428,7 @@ ENV AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} \
# Copy all scripts required for installation - changing any of those should lead to
# rebuilding from here
-COPY --from=scripts common.sh install_packaging_tools.sh \
- install_airflow_dependencies_from_branch_tip.sh create_prod_venv.sh /scripts/docker/
+COPY --from=scripts common.sh install_packaging_tools.sh create_prod_venv.sh /scripts/docker/
# We can set this value to true in case we want to install .whl/.tar.gz packages placed in the
# docker-context-files folder. This can be done for both additional packages you want to install
@@ -1536,13 +1458,7 @@ ENV AIRFLOW_CI_BUILD_EPOCH=${AIRFLOW_CI_BUILD_EPOCH}
# By default PIP installs everything to ~/.local and it's also treated as VIRTUALENV
ENV VIRTUAL_ENV="${AIRFLOW_USER_HOME_DIR}/.local"
-RUN bash /scripts/docker/install_packaging_tools.sh; \
- bash /scripts/docker/create_prod_venv.sh; \
- if [[ ${AIRFLOW_PRE_CACHED_PIP_PACKAGES} == "true" && \
- ${INSTALL_PACKAGES_FROM_CONTEXT} == "false" && \
- ${UPGRADE_INVALIDATION_STRING} == "" ]]; then \
- bash /scripts/docker/install_airflow_dependencies_from_branch_tip.sh; \
- fi
+RUN bash /scripts/docker/install_packaging_tools.sh; bash /scripts/docker/create_prod_venv.sh
COPY --chown=airflow:0 ${AIRFLOW_SOURCES_FROM} ${AIRFLOW_SOURCES_TO}
@@ -1566,10 +1482,10 @@ COPY --from=scripts install_from_docker_context_files.sh install_airflow.sh \
# an incorrect architecture.
ARG TARGETARCH
# Value to be able to easily change cache id and therefore use a bare new cache
-ARG PIP_CACHE_EPOCH="9"
+ARG DEPENDENCY_CACHE_EPOCH="9"
# hadolint ignore=SC2086, SC2010, DL3042
-RUN --mount=type=cache,id=$PYTHON_BASE_IMAGE-$AIRFLOW_PIP_VERSION-$TARGETARCH-$PIP_CACHE_EPOCH,target=/tmp/.cache/pip,uid=${AIRFLOW_UID} \
+RUN --mount=type=cache,id=prod-$TARGETARCH-$DEPENDENCY_CACHE_EPOCH,target=/tmp/.cache/,uid=${AIRFLOW_UID} \
if [[ ${INSTALL_PACKAGES_FROM_CONTEXT} == "true" ]]; then \
bash /scripts/docker/install_from_docker_context_files.sh; \
fi; \
@@ -1589,7 +1505,7 @@ RUN --mount=type=cache,id=$PYTHON_BASE_IMAGE-$AIRFLOW_PIP_VERSION-$TARGETARCH-$P
# during the build additionally to whatever has been installed so far. It is recommended that
# the requirements.txt contains only dependencies with == version specification
# hadolint ignore=DL3042
-RUN --mount=type=cache,id=additional-requirements-$PYTHON_BASE_IMAGE-$AIRFLOW_PIP_VERSION-$TARGETARCH-$PIP_CACHE_EPOCH,target=/tmp/.cache/pip,uid=${AIRFLOW_UID} \
+RUN --mount=type=cache,id=prod-$TARGETARCH-$DEPENDENCY_CACHE_EPOCH,target=/tmp/.cache/,uid=${AIRFLOW_UID} \
if [[ -f /docker-context-files/requirements.txt ]]; then \
pip install -r /docker-context-files/requirements.txt; \
fi
@@ -1617,7 +1533,9 @@ ARG PYTHON_BASE_IMAGE
ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE} \
# Make sure noninteractive debian install is used and language variables set
DEBIAN_FRONTEND=noninteractive LANGUAGE=C.UTF-8 LANG=C.UTF-8 LC_ALL=C.UTF-8 \
- LC_CTYPE=C.UTF-8 LC_MESSAGES=C.UTF-8 LD_LIBRARY_PATH=/usr/local/lib
+ LC_CTYPE=C.UTF-8 LC_MESSAGES=C.UTF-8 LD_LIBRARY_PATH=/usr/local/lib \
+ PIP_CACHE_DIR=/tmp/.cache/pip \
+ UV_CACHE_DIR=/tmp/.cache/uv
ARG RUNTIME_APT_DEPS=""
ARG ADDITIONAL_RUNTIME_APT_DEPS=""
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 16d3b29cf47..1c291893937 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -363,85 +363,6 @@ common::show_packaging_tool_version_and_location
common::install_packaging_tools
EOF
-# The content below is automatically copied from scripts/docker/install_airflow_dependencies_from_branch_tip.sh
-COPY <<"EOF" /install_airflow_dependencies_from_branch_tip.sh
-#!/usr/bin/env bash
-
-. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
-
-: "${AIRFLOW_REPO:?Should be set}"
-: "${AIRFLOW_BRANCH:?Should be set}"
-: "${INSTALL_MYSQL_CLIENT:?Should be true or false}"
-: "${INSTALL_POSTGRES_CLIENT:?Should be true or false}"
-
-function install_airflow_dependencies_from_branch_tip() {
- echo
- echo "${COLOR_BLUE}Installing airflow from ${AIRFLOW_BRANCH}. It is used to cache dependencies${COLOR_RESET}"
- echo
- if [[ ${INSTALL_MYSQL_CLIENT} != "true" ]]; then
- AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS/mysql,}
- fi
- if [[ ${INSTALL_POSTGRES_CLIENT} != "true" ]]; then
- AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS/postgres,}
- fi
- local TEMP_AIRFLOW_DIR
- TEMP_AIRFLOW_DIR=$(mktemp -d)
- # Install latest set of dependencies - without constraints. This is to download a "base" set of
- # dependencies that we can cache and reuse when installing airflow using constraints and latest
- # pyproject.toml in the next step (when we install regular airflow).
- set -x
- curl -fsSL "https://github.com/${AIRFLOW_REPO}/archive/${AIRFLOW_BRANCH}.tar.gz" | \
- tar xz -C "${TEMP_AIRFLOW_DIR}" --strip 1
- # Make sure editable dependencies are calculated when devel-ci dependencies are installed
- ${PACKAGING_TOOL_CMD} install ${EXTRA_INSTALL_FLAGS} ${ADDITIONAL_PIP_INSTALL_FLAGS} \
- --editable "${TEMP_AIRFLOW_DIR}[${AIRFLOW_EXTRAS}]"
- set +x
- common::install_packaging_tools
- set -x
- echo "${COLOR_BLUE}Uninstalling providers. Dependencies remain${COLOR_RESET}"
- # Uninstall airflow and providers to keep only the dependencies. In the future when
- # planned https://github.com/pypa/pip/issues/11440 is implemented in pip we might be able to use this
- # flag and skip the remove step.
- pip freeze | grep apache-airflow-providers | xargs ${PACKAGING_TOOL_CMD} uninstall ${EXTRA_UNINSTALL_FLAGS} || true
- set +x
- echo
- echo "${COLOR_BLUE}Uninstalling just airflow. Dependencies remain. Now target airflow can be reinstalled using mostly cached dependencies${COLOR_RESET}"
- echo
- set +x
- ${PACKAGING_TOOL_CMD} uninstall ${EXTRA_UNINSTALL_FLAGS} apache-airflow
- rm -rf "${TEMP_AIRFLOW_DIR}"
- set -x
- # If you want to make sure dependency is removed from cache in your PR when you removed it from
- # pyproject.toml - please add your dependency here as a list of strings
- # for example:
- # DEPENDENCIES_TO_REMOVE=("package_a" "package_b")
- # Once your PR is merged, you should make a follow-up PR to remove it from this list
- # and increase the AIRFLOW_CI_BUILD_EPOCH in Dockerfile.ci to make sure your cache is rebuilt.
- local DEPENDENCIES_TO_REMOVE
- # IMPORTANT!! Make sure to increase AIRFLOW_CI_BUILD_EPOCH in Dockerfile.ci when you remove a dependency from that list
- DEPENDENCIES_TO_REMOVE=()
- if [[ "${DEPENDENCIES_TO_REMOVE[*]}" != "" ]]; then
- echo
- echo "${COLOR_BLUE}Uninstalling just removed dependencies (temporary until cache refreshes)${COLOR_RESET}"
- echo "${COLOR_BLUE}Dependencies to uninstall: ${DEPENDENCIES_TO_REMOVE[*]}${COLOR_RESET}"
- echo
- set +x
- ${PACKAGING_TOOL_CMD} uninstall "${DEPENDENCIES_TO_REMOVE[@]}" || true
- set -x
- # make sure that the dependency is not needed by something else
- pip check
- fi
-}
-
-common::get_colors
-common::get_packaging_tool
-common::get_airflow_version_specification
-common::get_constraints_location
-common::show_packaging_tool_version_and_location
-
-install_airflow_dependencies_from_branch_tip
-EOF
-
# The content below is automatically copied from scripts/docker/common.sh
COPY <<"EOF" /common.sh
#!/usr/bin/env bash
@@ -627,35 +548,6 @@ function common::import_trusted_gpg() {
}
EOF
-# The content below is automatically copied from scripts/docker/install_pipx_tools.sh
-COPY <<"EOF" /install_pipx_tools.sh
-#!/usr/bin/env bash
-. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
-
-function install_pipx_tools() {
- echo
- echo "${COLOR_BLUE}Installing pipx tools${COLOR_RESET}"
- echo
- # Make sure PIPX is installed in latest version
- ${PACKAGING_TOOL_CMD} install ${EXTRA_INSTALL_FLAGS} --upgrade "pipx>=1.2.1"
- if [[ $(uname -m) != "aarch64" ]]; then
- # Do not install mssql-cli for ARM
- # Install all the tools we need available in command line but without impacting the current environment
- pipx install mssql-cli
-
- # Unfortunately mssql-cli installed by `pipx` does not work out of the box because it uses
- # its own execution bash script which is not compliant with the auto-activation of
- # pipx venvs - we need to manually patch Python executable in the script to fix it: ¯\_(ツ)_/¯
- sed "s/python /\/root\/\.local\/pipx\/venvs\/mssql-cli\/bin\/python /" -i /root/.local/bin/mssql-cli
- fi
-}
-
-common::get_colors
-common::get_packaging_tool
-
-install_pipx_tools
-EOF
-
# The content below is automatically copied from scripts/docker/install_airflow.sh
COPY <<"EOF" /install_airflow.sh
#!/usr/bin/env bash
@@ -1207,7 +1099,7 @@ ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow"
# NOTE! When you want to make sure dependencies are installed from scratch in your PR after removing
# some dependencies, you also need to set "disable image cache" in your PR to make sure the image is
# not built using the "main" version of those dependencies.
-ARG DEPENDENCIES_EPOCH_NUMBER="13"
+ARG DEPENDENCIES_EPOCH_NUMBER="14"
# Make sure noninteractive debian install is used and language variables set
ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE} \
@@ -1216,7 +1108,10 @@ ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE} \
DEPENDENCIES_EPOCH_NUMBER=${DEPENDENCIES_EPOCH_NUMBER} \
INSTALL_MYSQL_CLIENT="true" \
INSTALL_MSSQL_CLIENT="true" \
- INSTALL_POSTGRES_CLIENT="true"
+ INSTALL_POSTGRES_CLIENT="true" \
+ PIP_CACHE_DIR=/root/.cache/pip \
+ UV_CACHE_DIR=/root/.cache/uv
+
RUN echo "Base image version: ${PYTHON_BASE_IMAGE}"
@@ -1268,7 +1163,7 @@ RUN bash /scripts/docker/install_mysql.sh prod \
&& chmod 0440 /etc/sudoers.d/airflow
# Install Helm
-ARG HELM_VERSION="v3.15.3"
+ARG HELM_VERSION="v3.16.4"
RUN SYSTEM=$(uname -s | tr '[:upper:]' '[:lower:]') \
&& PLATFORM=$([ "$(uname -m)" = "aarch64" ] && echo "arm64" || echo "amd64" ) \
@@ -1296,15 +1191,7 @@ ARG DEFAULT_CONSTRAINTS_BRANCH="constraints-main"
# By changing the epoch we can force reinstalling Airflow and pip all dependencies
# It can also be overwritten manually by setting the AIRFLOW_CI_BUILD_EPOCH environment variable.
ARG AIRFLOW_CI_BUILD_EPOCH="10"
-ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="true"
-ARG AIRFLOW_PIP_VERSION=24.3.1
-ARG AIRFLOW_UV_VERSION=0.5.11
-ARG AIRFLOW_USE_UV="true"
# Setup PIP
-# By default PIP install run without cache to make image smaller
-ARG PIP_NO_CACHE_DIR="true"
-# By default UV install run without cache to make image smaller
-ARG UV_NO_CACHE="true"
ARG UV_HTTP_TIMEOUT="300"
# By default PIP has progress bar but you can disable it.
ARG PIP_PROGRESS_BAR="on"
@@ -1321,8 +1208,6 @@ ARG AIRFLOW_VERSION=""
# Additional PIP flags passed to all pip install commands except reinstalling pip itself
ARG ADDITIONAL_PIP_INSTALL_FLAGS=""
-ARG AIRFLOW_PIP_VERSION=24.3.1
-ARG AIRFLOW_UV_VERSION=0.5.11
ARG AIRFLOW_USE_UV="true"
ENV AIRFLOW_REPO=${AIRFLOW_REPO}\
@@ -1334,7 +1219,6 @@ ENV AIRFLOW_REPO=${AIRFLOW_REPO}\
AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION} \
DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH} \
AIRFLOW_CI_BUILD_EPOCH=${AIRFLOW_CI_BUILD_EPOCH} \
- AIRFLOW_PRE_CACHED_PIP_PACKAGES=${AIRFLOW_PRE_CACHED_PIP_PACKAGES} \
AIRFLOW_VERSION=${AIRFLOW_VERSION} \
AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} \
AIRFLOW_UV_VERSION=${AIRFLOW_UV_VERSION} \
@@ -1346,9 +1230,7 @@ ENV AIRFLOW_REPO=${AIRFLOW_REPO}\
INSTALL_POSTGRES_CLIENT="true" \
AIRFLOW_INSTALLATION_METHOD="." \
AIRFLOW_VERSION_SPECIFICATION="" \
- PIP_NO_CACHE_DIR=${PIP_NO_CACHE_DIR} \
PIP_PROGRESS_BAR=${PIP_PROGRESS_BAR} \
- UV_NO_CACHE=${UV_NO_CACHE} \
ADDITIONAL_PIP_INSTALL_FLAGS=${ADDITIONAL_PIP_INSTALL_FLAGS} \
CASS_DRIVER_BUILD_CONCURRENCY=${CASS_DRIVER_BUILD_CONCURRENCY} \
CASS_DRIVER_NO_CYTHON=${CASS_DRIVER_NO_CYTHON}
@@ -1357,42 +1239,51 @@ RUN echo "Airflow version: ${AIRFLOW_VERSION}"
# Copy all scripts required for installation - changing any of those should lead to
# rebuilding from here
-COPY --from=scripts install_packaging_tools.sh install_airflow_dependencies_from_branch_tip.sh \
- common.sh /scripts/docker/
+COPY --from=scripts common.sh install_packaging_tools.sh install_additional_dependencies.sh /scripts/docker/
# We are first creating a venv where all python packages and .so binaries needed by those are
# installed.
-# In case of CI builds we want to pre-install main version of airflow dependencies so that
-# We do not have to always reinstall it from the scratch.
-# And is automatically reinstalled from the scratch every time patch release of python gets released
-# The Airflow and providers are uninstalled, only dependencies remain.
-# the cache is only used when "upgrade to newer dependencies" is not set to automatically
-# account for removed dependencies (we do not install them in the first place)
-RUN bash /scripts/docker/install_packaging_tools.sh; \
- if [[ ${AIRFLOW_PRE_CACHED_PIP_PACKAGES} == "true" ]]; then \
- bash /scripts/docker/install_airflow_dependencies_from_branch_tip.sh; \
- fi
+
+# Here we fix the versions so all subsequent commands will use the versions
+# from the sources
+# You can swap comments between those two args to test pip from the main version
+# When you attempt to test if the version of `pip` from specified branch works for our builds
+# Also use `force pip` label on your PR to swap all places we use `uv` to `pip`
+ARG AIRFLOW_PIP_VERSION=24.3.1
+# ARG AIRFLOW_PIP_VERSION="git+https://github.com/pypa/pip.git@main"
+ARG AIRFLOW_UV_VERSION=0.5.14
+# TODO(potiuk): automate with upgrade check (possibly)
+ARG AIRFLOW_PRE_COMMIT_VERSION="4.0.1"
+ARG AIRFLOW_PRE_COMMIT_UV_VERSION="4.1.4"
+
+ENV AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} \
+ AIRFLOW_UV_VERSION=${AIRFLOW_UV_VERSION} \
+ # This is needed since we are using cache mounted from the host
+ UV_LINK_MODE=copy \
+ AIRFLOW_PRE_COMMIT_VERSION=${AIRFLOW_PRE_COMMIT_VERSION}
# The PATH is needed for PIPX to find the tools installed
ENV PATH="/root/.local/bin:${PATH}"
-COPY --from=scripts install_pipx_tools.sh /scripts/docker/
+# Useful for creating a cache id based on the underlying architecture, preventing the use of cached python packages from
+# an incorrect architecture.
+ARG TARGETARCH
+# Value to be able to easily change cache id and therefore use a bare new cache
+ARG DEPENDENCY_CACHE_EPOCH="0"
# Install useful command line tools in their own virtualenv so that they do not clash with
-# dependencies installed in Airflow
-RUN bash /scripts/docker/install_pipx_tools.sh
-
-# Airflow sources change frequently but dependency configuration won't change that often
-# We copy pyproject.toml and other files needed to perform setup of dependencies
-# So in case pyproject.toml changes we can install latest dependencies required.
-COPY pyproject.toml ${AIRFLOW_SOURCES}/pyproject.toml
-COPY airflow/__init__.py ${AIRFLOW_SOURCES}/airflow/
-COPY generated/* ${AIRFLOW_SOURCES}/generated/
-COPY constraints/* ${AIRFLOW_SOURCES}/constraints/
-COPY LICENSE ${AIRFLOW_SOURCES}/LICENSE
-COPY hatch_build.py ${AIRFLOW_SOURCES}/
+# dependencies installed in Airflow also reinstall PIP and UV to make sure they are installed
+# in the version specified above
+RUN bash /scripts/docker/install_packaging_tools.sh
+
COPY --from=scripts install_airflow.sh /scripts/docker/
+# We can copy everything here. The Context is filtered by dockerignore. This makes sure we are not
+# copying over stuff that is accidentally generated or that we do not need (such as egg-info)
+# if you want to add something that is missing and you expect to see it in the image you can
+# add it with ! in .dockerignore next to the airflow, test etc. directories there
+COPY . ${AIRFLOW_SOURCES}/
+
# Those are additional constraints that are needed for some extras but we do not want to
# force them on the main Airflow package. Currently we need no extra limits as PIP 23.1+ has much better
# dependency resolution and we do not need to limit the versions of the dependencies
@@ -1411,36 +1302,30 @@ ENV EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS=${EAGER_UPGRADE_ADDITIONAL_REQUIREMENT
# Usually we will install versions based on the dependencies in pyproject.toml and upgraded only if needed.
# But in cron job we will install latest versions matching pyproject.toml to see if there is no breaking change
# and push the constraints if everything is successful
-RUN bash /scripts/docker/install_airflow.sh
-
-COPY --from=scripts entrypoint_ci.sh /entrypoint
-COPY --from=scripts entrypoint_exec.sh /entrypoint-exec
-RUN chmod a+x /entrypoint /entrypoint-exec
+RUN --mount=type=cache,id=ci-$TARGETARCH-$DEPENDENCY_CACHE_EPOCH,target=/root/.cache/ bash /scripts/docker/install_airflow.sh
COPY --from=scripts install_packaging_tools.sh install_additional_dependencies.sh /scripts/docker/
-# Additional python deps to install
ARG ADDITIONAL_PYTHON_DEPS=""
-RUN bash /scripts/docker/install_packaging_tools.sh; \
+ENV ADDITIONAL_PYTHON_DEPS=${ADDITIONAL_PYTHON_DEPS}
+
+RUN --mount=type=cache,id=ci-$TARGETARCH-$DEPENDENCY_CACHE_EPOCH,target=/root/.cache/ \
+ bash /scripts/docker/install_packaging_tools.sh; \
if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
bash /scripts/docker/install_additional_dependencies.sh; \
fi
-# Install autocomplete for airflow
-RUN if command -v airflow; then \
- register-python-argcomplete airflow >> ~/.bashrc ; \
- fi
-
-# Install autocomplete for Kubectl
-RUN echo "source /etc/bash_completion" >> ~/.bashrc
+COPY --from=scripts entrypoint_ci.sh /entrypoint
+COPY --from=scripts entrypoint_exec.sh /entrypoint-exec
+RUN chmod a+x /entrypoint /entrypoint-exec
-# We can copy everything here. The Context is filtered by dockerignore. This makes sure we are not
-# copying over stuff that is accidentally generated or that we do not need (such as egg-info)
-# if you want to add something that is missing and you expect to see it in the image you can
-# add it with ! in .dockerignore next to the airflow, test etc. directories there
-COPY . ${AIRFLOW_SOURCES}/
+# Install autocomplete for airflow and kubectl
+RUN if command -v airflow; then \
+ register-python-argcomplete airflow >> ~/.bashrc ; \
+ fi; \
+ echo "source /etc/bash_completion" >> ~/.bashrc
WORKDIR ${AIRFLOW_SOURCES}
@@ -1451,7 +1336,13 @@ ARG AIRFLOW_IMAGE_DATE_CREATED
ENV PATH="/files/bin/:/opt/airflow/scripts/in_container/bin/:${PATH}" \
GUNICORN_CMD_ARGS="--worker-tmp-dir /dev/shm/" \
BUILD_ID=${BUILD_ID} \
- COMMIT_SHA=${COMMIT_SHA}
+ COMMIT_SHA=${COMMIT_SHA} \
+ # When we enter the image, the /root/.cache is not mounted from temporary mount cache.
+ # We do not want to share the cache from host to avoid all kinds of problems where cache
+ # is different with different platforms / python versions. We want to have a clean cache
+ # in the image - and in this case /root/.cache is on the same filesystem as the installed packages.
+ # so we can go back to the default link mode being hardlink.
+ UV_LINK_MODE=hardlink
# Link dumb-init for backwards compatibility (so that older images also work)
RUN ln -sf /usr/bin/dumb-init /usr/local/bin/dumb-init
diff --git a/contributing-docs/08_static_code_checks.rst b/contributing-docs/08_static_code_checks.rst
index 211ee7c9c34..246ca95d8c3 100644
--- a/contributing-docs/08_static_code_checks.rst
+++ b/contributing-docs/08_static_code_checks.rst
@@ -113,282 +113,257 @@ require Breeze Docker image to be built locally.
.. BEGIN AUTO-GENERATED STATIC CHECK LIST
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| ID | Description | Image |
-+===========================================================+==============================================================+=========+
-| bandit | bandit | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| blacken-docs | Run black on Python code blocks in documentation files | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-aiobotocore-optional | Check if aiobotocore is an optional dependency only | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-airflow-k8s-not-used | Check airflow.kubernetes imports are not used | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-airflow-provider-compatibility | Check compatibility of Providers with Airflow | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-airflow-providers-bug-report-template | Check airflow-bug-report provider list is sorted/unique | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-apache-license-rat | Check if licenses are OK for Apache | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-base-operator-partial-arguments | Check BaseOperator and partial() arguments | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-base-operator-usage | * Check BaseOperator core imports | |
-| | * Check BaseOperatorLink core imports | |
-| | * Check BaseOperator[Link] other imports | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-boring-cyborg-configuration | Checks for Boring Cyborg configuration consistency | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-breeze-top-dependencies-limited | Breeze should have small number of top-level dependencies | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-builtin-literals | Require literal syntax when initializing builtin types | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-changelog-format | Check changelog format | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-changelog-has-no-duplicates | Check changelogs for duplicate entries | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-cncf-k8s-only-for-executors | Check cncf.kubernetes imports used for executors only | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-code-deprecations | Check deprecations categories in decorators | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-common-compat-used-for-openlineage | Check common.compat is used for OL deprecated classes | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-compat-cache-on-methods | Check that compat cache do not use on class methods | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-core-deprecation-classes | Verify usage of Airflow deprecation classes in core | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-daysago-import-from-utils | Make sure days_ago is imported from airflow.utils.dates | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-decorated-operator-implements-custom-name | Check @task decorator implements custom_operator_name | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-deferrable-default | Check and fix default value of default_deferrable | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-docstring-param-types | Check that docstrings do not specify param types | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-example-dags-urls | Check that example dags url include provider versions | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-executables-have-shebangs | Check that executables have shebang | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-extra-packages-references | Checks setup extra packages | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-extras-order | Check order of extras in Dockerfile | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-fab-migrations | Check no migration is done on FAB related table | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-for-inclusive-language | Check for language that we do not accept as community | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-get-lineage-collector-providers | Check providers import hook lineage code from compat | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-google-re2-as-dependency | Check google-re2 is declared as dependency when needed | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-hatch-build-order | Check order of dependencies in hatch_build.py | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-hooks-apply | Check if all hooks apply to the repository | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-incorrect-use-of-LoggingMixin | Make sure LoggingMixin is not used alone | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-init-decorator-arguments | Check model __init__ and decorator arguments are in sync | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-integrations-list-consistent | Sync integrations list with docs | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-lazy-logging | Check that all logging methods are lazy | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-links-to-example-dags-do-not-use-hardcoded-versions | Verify example dags do not use hard-coded version numbers | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-merge-conflict | Check that merge conflicts are not being committed | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-min-python-version | Check minimum Python version | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-newsfragments-are-valid | Check newsfragments are valid | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-no-airflow-deprecation-in-providers | Do not use DeprecationWarning in providers | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-no-providers-in-core-examples | No providers imports in core example DAGs | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-only-new-session-with-provide-session | Check NEW_SESSION is only used with @provide_session | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-persist-credentials-disabled-in-github-workflows | Check that workflow files have persist-credentials disabled | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-pre-commit-information-consistent | Validate hook IDs & names and sync with docs | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-provide-create-sessions-imports | Check provide_session and create_session imports | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-provider-docs-valid | Validate provider doc files | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-provider-yaml-valid | Validate provider.yaml files | * |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-providers-init-file-missing | Provider init file is missing | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-providers-subpackages-init-file-exist | Provider subpackage init files are there | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-pydevd-left-in-code | Check for pydevd debug statements accidentally left | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-revision-heads-map | Check that the REVISION_HEADS_MAP is up-to-date | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-safe-filter-usage-in-html | Don't use safe in templates | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-sql-dependency-common-data-structure | Check dependency of SQL Providers with common data structure | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-start-date-not-used-in-defaults | start_date not to be defined in default_args in example_dags | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-system-tests-present | Check if system tests have required segments of code | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-system-tests-tocs | Check that system tests is properly added | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-taskinstance-tis-attrs | Check that TI and TIS have the same attributes | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-template-context-variable-in-sync | Check all template context variable references are in sync | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-tests-in-the-right-folders | Check if tests are in the right folders | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-tests-unittest-testcase | Check that unit tests do not inherit from unittest.TestCase | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-urlparse-usage-in-code | Don't use urlparse in code | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-usage-of-re2-over-re | Use re2 module instead of re | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| check-xml | Check XML files with xmllint | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| codespell | Run codespell to check for common misspellings in files | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| compile-www-assets | Compile www assets (manual) | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| compile-www-assets-dev | Compile www assets in dev mode (manual) | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| create-missing-init-py-files-tests | Create missing init.py files in tests | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| debug-statements | Detect accidentally committed debug statements | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| detect-private-key | Detect if private key is added to the repository | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| doctoc | Add TOC for Markdown and RST files | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| end-of-file-fixer | Make sure that there is an empty line at the end | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| fix-encoding-pragma | Remove encoding header from Python files | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| flynt | Run flynt string format converter for Python | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| generate-airflow-diagrams | Generate airflow diagrams | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| generate-pypi-readme | Generate PyPI README | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| identity | Print input to the static check hooks for troubleshooting | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| insert-license | * Add license for all SQL files | |
-| | * Add license for all RST files | |
-| | * Add license for all CSS/JS/JSX/PUML/TS/TSX files | |
-| | * Add license for all JINJA template files | |
-| | * Add license for all Shell files | |
-| | * Add license for all toml files | |
-| | * Add license for all Python files | |
-| | * Add license for all XML files | |
-| | * Add license for all Helm template files | |
-| | * Add license for all YAML files except Helm templates | |
-| | * Add license for all Markdown files | |
-| | * Add license for all other files | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| kubeconform | Kubeconform check on our helm chart | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| lint-chart-schema | Lint chart/values.schema.json file | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| lint-css | stylelint | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| lint-dockerfile | Lint Dockerfile | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| lint-helm-chart | Lint Helm Chart | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| lint-json-schema | * Lint JSON Schema files with JSON Schema | |
-| | * Lint NodePort Service with JSON Schema | |
-| | * Lint Docker compose files with JSON Schema | |
-| | * Lint chart/values.schema.json file with JSON Schema | |
-| | * Lint chart/values.yaml file with JSON Schema | |
-| | * Lint config_templates/config.yml file with JSON Schema | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| lint-markdown | Run markdownlint | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| lint-openapi | * Lint OpenAPI using spectral | |
-| | * Lint OpenAPI using openapi-spec-validator | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| mixed-line-ending | Detect if mixed line ending is used (\r vs. \r\n) | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| mypy-airflow | * Run mypy for airflow | * |
-| | * Run mypy for airflow (manual) | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| mypy-dev | * Run mypy for dev | * |
-| | * Run mypy for dev (manual) | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| mypy-docs | * Run mypy for /docs/ folder | * |
-| | * Run mypy for /docs/ folder (manual) | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| mypy-providers | * Run mypy for providers | * |
-| | * Run mypy for providers (manual) | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| pretty-format-json | Format JSON files | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| pylint | pylint | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| python-no-log-warn | Check if there are no deprecate log warn | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| replace-bad-characters | Replace bad characters | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| rst-backticks | Check if RST files use double backticks for code | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| ruff | Run 'ruff' for extremely fast Python linting | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| ruff-format | Run 'ruff format' for extremely fast Python formatting | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| shellcheck | Check Shell scripts syntax correctness | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| trailing-whitespace | Remove trailing whitespace at end of line | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| ts-compile-format-lint-www | TS types generation / ESLint / Prettier against UI files | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| update-black-version | Update black versions everywhere (manual) | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| update-breeze-cmd-output | Update output of breeze commands in Breeze documentation | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| update-breeze-readme-config-hash | Update Breeze README.md with config files hash | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| update-build-dependencies | Update build-dependencies to latest (manual) | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| update-chart-dependencies | Update chart dependencies to latest (manual) | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| update-common-sql-api-stubs | Check and update common.sql API stubs | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| update-er-diagram | Update ER diagram | * |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| update-extras | Update extras in documentation | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| update-in-the-wild-to-be-sorted | Sort INTHEWILD.md alphabetically | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| update-inlined-dockerfile-scripts | Inline Dockerfile and Dockerfile.ci scripts | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| update-installed-providers-to-be-sorted | Sort alphabetically and uniquify installed_providers.txt | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| update-installers | Update installers to latest (manual) | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| update-local-yml-file | Update mounts in the local yml file | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| update-migration-references | Update migration ref doc | * |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| update-openapi-spec-tags-to-be-sorted | Sort alphabetically openapi spec tags | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| update-providers-dependencies | Update dependencies for provider packages | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| update-reproducible-source-date-epoch | Update Source Date Epoch for reproducible builds | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| update-spelling-wordlist-to-be-sorted | Sort alphabetically and uniquify spelling_wordlist.txt | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| update-supported-versions | Updates supported versions in documentation | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| update-vendored-in-k8s-json-schema | Vendor k8s definitions into values.schema.json | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| update-version | Update version to the latest version in the documentation | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| validate-operators-init | Prevent templated field logic checks in operators' __init__ | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
-| yamllint | Check YAML files with yamllint | |
-+-----------------------------------------------------------+--------------------------------------------------------------+---------+
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| ID | Description | Image |
++===========================================================+========================================================+=========+
+| bandit | bandit | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| blacken-docs | Run black on docs | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-aiobotocore-optional | Check if aiobotocore is an optional dependency only | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-airflow-k8s-not-used | Check airflow.kubernetes imports are not used | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-apache-license-rat | Check if licenses are OK for Apache | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-base-operator-partial-arguments | Check BaseOperator and partial() arguments | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-base-operator-usage | * Check BaseOperator core imports | |
+| | * Check BaseOperatorLink core imports | |
+| | * Check BaseOperator[Link] other imports | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-boring-cyborg-configuration | Checks for Boring Cyborg configuration consistency | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-breeze-top-dependencies-limited | Check top-level breeze deps | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-builtin-literals | Require literal syntax when initializing builtins | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-changelog-format | Check changelog format | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-changelog-has-no-duplicates | Check changelogs for duplicate entries | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-code-deprecations | Check deprecations categories in decorators | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-common-compat-used-for-openlineage | Check common.compat is used for OL deprecated classes | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-core-deprecation-classes | Verify usage of Airflow deprecation classes in core | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-daysago-import-from-utils | days_ago imported from airflow.utils.dates | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-decorated-operator-implements-custom-name | Check @task decorator implements custom_operator_name | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-deferrable-default | Check and fix default value of default_deferrable | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-docstring-param-types | Check that docstrings do not specify param types | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-example-dags-urls | Check that example dags url include provider versions | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-executables-have-shebangs | Check that executables have shebang | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-extra-packages-references | Checks setup extra packages | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-extras-order | Check order of extras in Dockerfile | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-fab-migrations | Check no migration is done on FAB related table | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-for-inclusive-language | Check for language that we do not accept as community | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-get-lineage-collector-providers | Check providers import hook lineage code from compat | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-hatch-build-order | Check order of dependencies in hatch_build.py | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-hooks-apply | Check if all hooks apply to the repository | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-incorrect-use-of-LoggingMixin | Make sure LoggingMixin is not used alone | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-init-decorator-arguments | Sync model __init__ and decorator arguments | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-integrations-list-consistent | Sync integrations list with docs | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-lazy-logging | Check that all logging methods are lazy | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-links-to-example-dags-do-not-use-hardcoded-versions | Verify no hard-coded version in example dags | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-merge-conflict | Check that merge conflicts are not being committed | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-min-python-version | Check minimum Python version | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-newsfragments-are-valid | Check newsfragments are valid | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-no-airflow-deprecation-in-providers | Do not use DeprecationWarning in providers | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-no-providers-in-core-examples | No providers imports in core example DAGs | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-only-new-session-with-provide-session | Check NEW_SESSION is only used with @provide_session | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-persist-credentials-disabled-in-github-workflows | Check persistent creds in workflow files | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-pre-commit-information-consistent | Validate hook IDs & names and sync with docs | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-provide-create-sessions-imports | Check session util imports | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-provider-docs-valid | Validate provider doc files | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-provider-yaml-valid | Validate provider.yaml files | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-providers-subpackages-init-file-exist | Provider subpackage init files are there | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-pydevd-left-in-code | Check for pydevd debug statements accidentally left | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-safe-filter-usage-in-html | Don't use safe in templates | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-start-date-not-used-in-defaults | start_date not in default_args | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-system-tests-present | Check if system tests have required segments of code | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-system-tests-tocs | Check that system tests is properly added | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-taskinstance-tis-attrs | Check that TI and TIS have the same attributes | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-template-context-variable-in-sync | Sync template context variable refs | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-tests-in-the-right-folders | Check if tests are in the right folders | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-tests-unittest-testcase | Unit tests do not inherit from unittest.TestCase | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-urlparse-usage-in-code | Don't use urlparse in code | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-usage-of-re2-over-re | Use re2 module instead of re | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| check-xml | Check XML files with xmllint | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| codespell | Run codespell | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| compile-www-assets | Compile www assets (manual) | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| compile-www-assets-dev | Compile www assets in dev mode (manual) | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| create-missing-init-py-files-tests | Create missing init.py files in tests | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| debug-statements | Detect accidentally committed debug statements | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| detect-private-key | Detect if private key is added to the repository | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| doctoc | Add TOC for Markdown and RST files | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| end-of-file-fixer | Make sure that there is an empty line at the end | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| fix-encoding-pragma | Remove encoding header from Python files | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| flynt | Run flynt string format converter for Python | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| generate-airflow-diagrams | Generate airflow diagrams | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| generate-pypi-readme | Generate PyPI README | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| identity | Print checked files | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| insert-license | * Add license for all SQL files | |
+| | * Add license for all RST files | |
+| | * Add license for CSS/JS/JSX/PUML/TS/TSX | |
+| | * Add license for all JINJA template files | |
+| | * Add license for all Shell files | |
+| | * Add license for all toml files | |
+| | * Add license for all Python files | |
+| | * Add license for all XML files | |
+| | * Add license for all Helm template files | |
+| | * Add license for all YAML files except Helm templates | |
+| | * Add license for all Markdown files | |
+| | * Add license for all other files | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| kubeconform | Kubeconform check on our helm chart | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| lint-chart-schema | Lint chart/values.schema.json file | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| lint-css | stylelint | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| lint-dockerfile | Lint Dockerfile | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| lint-helm-chart | Lint Helm Chart | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| lint-json-schema | * Lint JSON Schema files | |
+| | * Lint NodePort Service | |
+| | * Lint Docker compose files | |
+| | * Lint chart/values.schema.json | |
+| | * Lint chart/values.yaml | |
+| | * Lint config_templates/config.yml | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| lint-markdown | Run markdownlint | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| lint-openapi | * Lint OpenAPI using spectral | |
+| | * Lint OpenAPI using openapi-spec-validator | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| mixed-line-ending | Detect if mixed line ending is used (\r vs. \r\n) | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| mypy-airflow | * Run mypy for airflow | |
+| | * Run mypy for airflow (manual) | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| mypy-docs | * Run mypy for /docs/ folder | |
+| | * Run mypy for /docs/ folder (manual) | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| mypy-providers | * Run mypy for providers | |
+| | * Run mypy for providers (manual) | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| pretty-format-json | Format JSON files | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| pylint | pylint | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| python-no-log-warn | Check if there are no deprecate log warn | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| replace-bad-characters | Replace bad characters | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| rst-backticks | Check if RST files use double backticks for code | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| ruff | Run 'ruff' for extremely fast Python linting | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| ruff-format | Run 'ruff format' | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| shellcheck | Check Shell scripts syntax correctness | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| trailing-whitespace | Remove trailing whitespace at end of line | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| ts-compile-format-lint-www | Compile / format / lint WWW | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| update-black-version | Update black versions everywhere (manual) | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| update-breeze-cmd-output | Update breeze docs | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| update-breeze-readme-config-hash | Update Breeze README.md with config files hash | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| update-chart-dependencies | Update chart dependencies to latest (manual) | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| update-er-diagram | Update ER diagram | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| update-in-the-wild-to-be-sorted | Sort INTHEWILD.md alphabetically | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| update-inlined-dockerfile-scripts | Inline Dockerfile and Dockerfile.ci scripts | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| update-installed-providers-to-be-sorted | Sort and uniquify installed_providers.txt | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| update-installers-and-pre-commit | Update installers and pre-commit to latest (manual) | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| update-local-yml-file | Update mounts in the local yml file | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| update-migration-references | Update migration ref doc | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| update-openapi-spec-tags-to-be-sorted | Sort alphabetically openapi spec tags | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| update-providers-dependencies | Update dependencies for provider packages | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| update-reproducible-source-date-epoch | Update Source Date Epoch for reproducible builds | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| update-spelling-wordlist-to-be-sorted | Sort spelling_wordlist.txt | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| update-supported-versions | Updates supported versions in documentation | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| update-vendored-in-k8s-json-schema | Vendor k8s definitions into values.schema.json | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| update-version | Update versions in docs | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| yamllint | Check YAML files with yamllint | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
+| zizmor | Run zizmor to check for github workflow syntax errors | |
++-----------------------------------------------------------+--------------------------------------------------------+---------+
.. END AUTO-GENERATED STATIC CHECK LIST
diff --git a/contributing-docs/testing/integration_tests.rst b/contributing-docs/testing/integration_tests.rst
index 322298d4f00..ea9dfb7e952 100644
--- a/contributing-docs/testing/integration_tests.rst
+++ b/contributing-docs/testing/integration_tests.rst
@@ -49,39 +49,41 @@ The following integrations are available:
.. BEGIN AUTO-GENERATED INTEGRATION LIST
-+--------------+----------------------------------------------------+
-| Identifier | Description |
-+==============+====================================================+
-| cassandra | Integration required for Cassandra hooks. |
-+--------------+----------------------------------------------------+
-| celery | Integration required for Celery executor tests. |
-+--------------+----------------------------------------------------+
-| drill | Integration required for drill operator and hook. |
-+--------------+----------------------------------------------------+
-| kafka | Integration required for Kafka hooks. |
-+--------------+----------------------------------------------------+
-| kerberos | Integration that provides Kerberos authentication. |
-+--------------+----------------------------------------------------+
-| mongo | Integration required for MongoDB hooks. |
-+--------------+----------------------------------------------------+
-| mssql | Integration required for mssql hooks. |
-+--------------+----------------------------------------------------+
-| openlineage | Integration required for Openlineage hooks. |
-+--------------+----------------------------------------------------+
-| otel | Integration required for OTEL/opentelemetry hooks. |
-+--------------+----------------------------------------------------+
-| pinot | Integration required for Apache Pinot hooks. |
-+--------------+----------------------------------------------------+
-| qdrant | Integration required for Qdrant tests. |
-+--------------+----------------------------------------------------+
-| redis | Integration required for Redis tests. |
-+--------------+----------------------------------------------------+
-| statsd | Integration required for Statsd hooks. |
-+--------------+----------------------------------------------------+
-| trino | Integration required for Trino hooks. |
-+--------------+----------------------------------------------------+
-| ydb | Integration required for YDB tests. |
-+--------------+----------------------------------------------------+
++--------------+-------------------------------------------------------+
+| Identifier | Description |
++==============+=======================================================+
+| cassandra | Integration required for Cassandra hooks. |
++--------------+-------------------------------------------------------+
+| celery | Integration required for Celery executor tests. |
++--------------+-------------------------------------------------------+
+| drill | Integration required for drill operator and hook. |
++--------------+-------------------------------------------------------+
+| kafka | Integration required for Kafka hooks. |
++--------------+-------------------------------------------------------+
+| kerberos | Integration that provides Kerberos authentication. |
++--------------+-------------------------------------------------------+
+| keycloak | Integration for manual testing of multi-team Airflow. |
++--------------+-------------------------------------------------------+
+| mongo | Integration required for MongoDB hooks. |
++--------------+-------------------------------------------------------+
+| mssql | Integration required for mssql hooks. |
++--------------+-------------------------------------------------------+
+| openlineage | Integration required for Openlineage hooks. |
++--------------+-------------------------------------------------------+
+| otel | Integration required for OTEL/opentelemetry hooks. |
++--------------+-------------------------------------------------------+
+| pinot | Integration required for Apache Pinot hooks. |
++--------------+-------------------------------------------------------+
+| qdrant | Integration required for Qdrant tests. |
++--------------+-------------------------------------------------------+
+| redis | Integration required for Redis tests. |
++--------------+-------------------------------------------------------+
+| statsd | Integration required for Statsd hooks. |
++--------------+-------------------------------------------------------+
+| trino | Integration required for Trino hooks. |
++--------------+-------------------------------------------------------+
+| ydb | Integration required for YDB tests. |
++--------------+-------------------------------------------------------+
.. END AUTO-GENERATED INTEGRATION LIST'
diff --git a/dev/breeze/README.md b/dev/breeze/README.md
index 5e5db1f26e8..cafa346cbaa 100644
--- a/dev/breeze/README.md
+++ b/dev/breeze/README.md
@@ -22,6 +22,7 @@
**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
- [Apache Airflow Breeze](#apache-airflow-breeze)
+- [Setting up development env for Breeze](#setting-up-development-env-for-breeze)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
@@ -34,27 +35,19 @@ for Airflow Development.
This package should never be installed in "production" mode. The `breeze` entrypoint will actually
fail if you do so. It is supposed to be installed only in [editable/development mode](https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#working-in-development-mode)
-directly from Airflow sources using `pipx` - usually with `--force` flag to account for re-installation
-that might often be needed if dependencies change during development.
+directly from Airflow sources using `uv tool` or `pipx` - usually with `--force` flag to account
+for re-installation that might often be needed if dependencies change during development.
```shell
-pipx install -e ./dev/breeze --force
+uv tool install -e ./dev/breeze --force
```
-NOTE! If you see below warning - it means that you hit [known issue](https://github.com/pypa/pipx/issues/1092)
-with `packaging` version 23.2
-⚠️ Ignoring --editable install option. pipx disallows it for anything but a local path,
-to avoid having to create a new src/ directory.
-
-The workaround is to downgrade packaging to 23.1 and re-running the `pipx install` command, for example
-by running `pip install "packaging<23.2"`.
+or
```shell
-pip install "packaging<23.2"
pipx install -e ./dev/breeze --force
```
-
You can read more about Breeze in the [documentation](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst)
This README file contains automatically generated hash of the `pyproject.toml` files that were
@@ -62,10 +55,79 @@ available when the package was installed. Since this file becomes part of the in
to detect automatically if any of the files have changed. If they did, the user will be warned to upgrade
their installations.
+Setting up development env for Breeze
+-------------------------------------
+
+> [!NOTE]
+> This section is for developers of Breeze. If you are a user of Breeze, you do not need to read this section.
+
+Breeze is actively developed by Airflow maintainers and contributors, Airflow is an active project
+and we are in the process of developing Airflow 3, so breeze requires a lot of adjustments to keep up
+the dev environment in sync with Airflow 3 development - this is also why it is part of the same
+repository as Airflow - because it needs to be closely synchronized with Airflow development.
+
+As of November 2024 Airflow switches to using `uv` as the main development environment for Airflow
+and for Breeze. So the instructions below are for setting up the development environment for Breeze
+using `uv`. However we are using only standard python packaging tools, so you can still use `pip` or
+`pipenv` or other build frontends to install Breeze, but we recommend using `uv` as it is the most
+convenient way to install, manage python packages and virtual environments.
+
+Unlike in Airflow, where we manage our own constraints, we use `uv` to manage requirements for Breeze
+and we use `uv` to lock the dependencies. This way we can ensure that the dependencies are always
+up-to-date and that the development environment is always consistent for different people. This is
+why Breeze's `uv.lock` is committed to the repository and is used to install the dependencies by
+default by Breeze. Here's how to install breeze with `uv`
+
+
+1. Install `uv` - see [uv documentation](https://docs.astral.sh/uv/getting-started/installation/)
+
+> [!IMPORTANT]
+> All the commands below should be executed while you are in `dev/breeze` directory of the Airflow repository.
+
+2. Create a new virtual environment for Breeze development:
+
+```shell
+uv venv
+```
+
+3. Synchronize Breeze dependencies with `uv` to the latest dependencies stored in uv.lock file:
+
+```shell
+uv sync
+```
+
+After syncing, the `.venv` directory will contain the virtual environment with all the dependencies
+installed - you can use that environment to develop Breeze - for example with your favourite IDE
+or text editor, you can also use `uv run` to run the scripts in the virtual environment.
+
+For example to run all tests in the virtual environment you can use:
+
+```shell
+uv run pytest
+```
+
+4. Add/remove dependencies with `uv`:
+
+```shell
+uv add <package>
+uv remove <package>
+```
+
+5. Update and lock the dependencies (after adding them or periodically to keep them up-to-date):
+
+```shell
+uv lock
+```
+
+Note that when you update dependencies/lock them you should commit the changes in `pyproject.toml` and `uv.lock`.
+
+See [uv documentation](https://docs.astral.sh/uv/getting-started/) for more details on using `uv`.
+
+
PLEASE DO NOT MODIFY THE HASH BELOW! IT IS AUTOMATICALLY UPDATED BY PRE-COMMIT.
---------------------------------------------------------------------------------------------------------
-Package config hash: b2e0a459155b5c685597fdf153589d13ce8730b312bccc5fb8810ced491730649239e07261bb9c7fdbe361a20d3c5497d0ca65dbd2964a2b74cf065d04f710fc
+Package config hash: 47876b91d3af997736722846114da2af8be13f35e7c2e00e5bc7c56bbc47cb2b9eafed53667a2752b0be5f753b10d79366e47c4fe79a06524155b53352f26f66
---------------------------------------------------------------------------------------------------------
diff --git a/dev/breeze/doc/01_installation.rst b/dev/breeze/doc/01_installation.rst
index aad8640e7f6..9c053bf280b 100644
--- a/dev/breeze/doc/01_installation.rst
+++ b/dev/breeze/doc/01_installation.rst
@@ -249,27 +249,31 @@ In case of disk space errors on macOS, increase the disk space available for Doc
Installation
============
+First, clone the Airflow repository, but make sure not to clone it into your home directory. Cloning it into your home directory will cause the following error:
+``Your Airflow sources are checked out in /Users/username/airflow, which is also your AIRFLOW_HOME where Airflow writes logs and database files. This setup is problematic because Airflow might overwrite or clean up your source code and .git repository.``
-Set your working directory to root of (this) cloned repository.
-Run this command to install Breeze (make sure to use ``-e`` flag):
+.. code-block:: bash
+
+ git clone https://github.com/apache/airflow.git
+
+Set your working directory to the root of this cloned repository.
.. code-block:: bash
- pipx install -e ./dev/breeze
+ cd airflow
-.. warning::
+Run this command to install Breeze (make sure to use ``-e`` flag) - you can choose ``uv`` (recommended) or
+``pipx``:
- If you see below warning - it means that you hit `known issue <https://github.com/pypa/pipx/issues/1092>`_
- with ``packaging`` version 23.2:
- ⚠️ Ignoring --editable install option. pipx disallows it for anything but a local path,
- to avoid having to create a new src/ directory.
- The workaround is to downgrade packaging to 23.1 and re-running the ``pipx install`` command.
+.. code-block:: bash
- .. code-block:: bash
+ uv tool install -e ./dev/breeze
- pip install "packaging<23.2"
- pipx install -e ./dev/breeze --force
+
+.. code-block:: bash
+
+ pipx install -e ./dev/breeze
.. note:: Note for Windows users
@@ -278,6 +282,12 @@ Run this command to install Breeze (make sure to use ``-e`` flag):
If you are on Windows, you should use Windows way to point to the ``dev/breeze`` sub-folder
of Airflow either as absolute or relative path. For example:
+ .. code-block:: bash
+
+ uv tool install -e dev\breeze
+
+ or
+
.. code-block:: bash
pipx install -e dev\breeze
@@ -317,8 +327,14 @@ that Breeze works on
.. warning:: Upgrading from earlier Python version
- If you used Breeze with Python 3.7 and when running it, it will complain that it needs Python 3.8. In this
- case you should force-reinstall Breeze with ``pipx``:
+ If you used Breeze with Python 3.8 and when running it, it will complain that it needs Python 3.9. In this
+ case you should force-reinstall Breeze with ``uv`` (or ``pipx``):
+
+ .. code-block:: bash
+
+ uv tool install --force -e ./dev/breeze
+
+ or
.. code-block:: bash
@@ -330,30 +346,35 @@ that Breeze works on
If you are on Windows, you should use Windows way to point to the ``dev/breeze`` sub-folder
of Airflow either as absolute or relative path. For example:
+ .. code-block:: bash
+
+ uv tool install --force -e dev\breeze
+
+ or
+
.. code-block:: bash
pipx install --force -e dev\breeze
- .. note:: creating pipx virtual env ``apache-airflow-breeze`` with a specific python version
- In ``pipx install -e ./dev/breeze`` or ``pipx install -e dev\breeze``, ``pipx`` uses default
- system python version to create virtual env for breeze.
- We can use a specific version by providing python executable in ``--python`` argument. For example:
+ .. note:: creating virtual env for ``apache-airflow-breeze`` with a specific python version
+ The ``uv tool install`` or ``pipx install`` use default system python version to create virtual
+ env for breeze. You can use a specific version by providing python version in ``uv`` or
+ python executable in ``pipx`` in ``--python``.
If you have breeze installed already with another Python version you can reinstall breeze with reinstall
command
.. code-block:: bash
- pipx reinstall --python /Users/airflow/.pyenv/versions/3.8.16/bin/python apache-airflow-breeze
+ uv tool install --python 3.9.16 ./dev/breeze --force
- Or you can uninstall breeze and install it with a specific python version:
+ or
.. code-block:: bash
- pipx uninstall apache-airflow-breeze
- pipx install -e ./dev/breeze --python /Users/airflow/.pyenv/versions/3.8.16/bin/python
+ pipx install -e ./dev/breeze --python /Users/airflow/.pyenv/versions/3.9.16/bin/python --force
Running Breeze for the first time
@@ -466,19 +487,18 @@ Automating breeze installation
------------------------------
Breeze on POSIX-compliant systems (Linux, MacOS) can be automatically installed by running the
-``scripts/tools/setup_breeze`` bash script. This includes checking and installing ``pipx``, setting up
+``scripts/tools/setup_breeze`` bash script. This includes checking and installing ``uv``, setting up
``breeze`` with it and setting up autocomplete.
Uninstalling Breeze
-------------------
-Since Breeze is installed with ``pipx``, with ``pipx list``, you can list the installed packages.
-Once you have the name of ``breeze`` package you can proceed to uninstall it.
+Since Breeze is installed with ``uv tool`` or ``pipx``, you need to use the appropriate tool to uninstall it.
.. code-block:: bash
- pipx list
+ uv tool uninstall apache-airflow-breeze
This will also remove breeze from the folder: ``${HOME}.local/bin/``
diff --git a/dev/breeze/doc/02_customizing.rst b/dev/breeze/doc/02_customizing.rst
index 291314abfc3..009b5a25149 100644
--- a/dev/breeze/doc/02_customizing.rst
+++ b/dev/breeze/doc/02_customizing.rst
@@ -61,6 +61,40 @@ so you can change it at any place, and run
inside container, to enable modified tmux configurations.
+Tmux tldr
+~~~~~~~~~
+
+In case you, like some Airflow core devs, are a tmux dummy, here are some tmux config entries
+that you may find helpful.
+
+.. code-block::
+
+ # if you like vi mode instead of emacs
+ set-window-option -g mode-keys vi
+
+ # will not clear the selection immediately
+ bind-key -T copy-mode-vi MouseDragEnd1Pane send-keys -X copy-selection-no-clear
+
+ # make it so ctrl+shift+arrow moves the focused pane
+ bind -T root C-S-Left select-pane -L
+ bind -T root C-S-Right select-pane -R
+ bind -T root C-S-Up select-pane -U
+ bind -T root C-S-Down select-pane -D
+
+Some helpful commands:
+
+ - ``ctrl-b + z``: zoom into selected pane
+ - ``ctrl-b + [``: enter copy mode
+
+To copy an entire pane:
+ - select the pane
+ - enter copy mode: ``ctrl-b + [``
+ - go to start: ``g``
+ - begin selection: ``space``
+ - extend selection to end: ``G``
+ - copy and clear selection: ``enter``
+
+
Additional tools in Breeze container
------------------------------------
diff --git a/dev/breeze/doc/03_developer_tasks.rst b/dev/breeze/doc/03_developer_tasks.rst
index 3ad8df4773b..ad1a4fe0a6f 100644
--- a/dev/breeze/doc/03_developer_tasks.rst
+++ b/dev/breeze/doc/03_developer_tasks.rst
@@ -34,12 +34,12 @@ You can use additional ``breeze`` flags to choose your environment. You can spec
version to use, and backend (the meta-data database). Thanks to that, with Breeze, you can recreate the same
environments as we have in matrix builds in the CI. See next chapter for backend selection.
-For example, you can choose to run Python 3.8 tests with MySQL as backend and with mysql version 8
+For example, you can choose to run Python 3.9 tests with MySQL as backend and with mysql version 8
as follows:
.. code-block:: bash
- breeze --python 3.8 --backend mysql --mysql-version 8.0
+ breeze --python 3.9 --backend mysql --mysql-version 8.0
.. note:: Note for Windows WSL2 users
@@ -55,7 +55,7 @@ Try adding ``--builder=default`` to your command. For example:
.. code-block:: bash
- breeze --builder=default --python 3.8 --backend mysql --mysql-version 8.0
+ breeze --builder=default --python 3.9 --backend mysql --mysql-version 8.0
The choices you make are persisted in the ``./.build/`` cache directory so that next time when you use the
``breeze`` script, it could use the values that were used previously. This way you do not have to specify
@@ -113,6 +113,7 @@ When you run Airflow Breeze, the following ports are automatically forwarded:
* 12322 -> forwarded to Airflow ssh server -> airflow:22
* 28080 -> forwarded to Airflow webserver -> airflow:8080
+ * 29091 -> forwarded to Airflow FastAPI API -> airflow:9091
* 25555 -> forwarded to Flower dashboard -> airflow:5555
* 25433 -> forwarded to Postgres database -> postgres:5432
* 23306 -> forwarded to MySQL database -> mysql:3306
@@ -125,6 +126,7 @@ You can connect to these ports/databases using:
* ssh connection for remote debugging: ssh -p 12322 airflow@127.0.0.1 pw: airflow
* Webserver: http://127.0.0.1:28080
+ * FastAPI API: http://127.0.0.1:29091
* Flower: http://127.0.0.1:25555
* Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow
* Mysql: jdbc:mysql://127.0.0.1:23306/airflow?user=root
@@ -154,6 +156,7 @@ You can change the used host port numbers by setting appropriate environment var
* ``SSH_PORT``
* ``WEBSERVER_HOST_PORT``
+* ``FASTAPI_API_HOST_PORT``
* ``POSTGRES_HOST_PORT``
* ``MYSQL_HOST_PORT``
* ``MSSQL_HOST_PORT``
@@ -328,7 +331,7 @@ When you are starting airflow from local sources, www asset compilation is autom
.. code-block:: bash
- breeze --python 3.8 --backend mysql start-airflow
+ breeze --python 3.9 --backend mysql start-airflow
You can also use it to start different executor.
@@ -341,7 +344,7 @@ You can also use it to start any released version of Airflow from ``PyPI`` with
.. code-block:: bash
- breeze start-airflow --python 3.8 --backend mysql --use-airflow-version 2.7.0
+ breeze start-airflow --python 3.9 --backend mysql --use-airflow-version 2.7.0
When you are installing version from PyPI, it's also possible to specify extras that should be used
when installing Airflow - you can provide several extras separated by coma - for example to install
@@ -397,6 +400,17 @@ command takes care about it. This is needed when you want to run webserver insid
:width: 100%
:alt: Breeze compile-www-assets
+Compiling ui assets
+--------------------
+
+Airflow webserver needs to prepare www assets - compiled with node and yarn. The ``compile-ui-assets``
+command takes care about it. This is needed when you want to run webserver inside of the breeze.
+
+.. image:: ./images/output_compile-ui-assets.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_compile-ui-assets.svg
+ :width: 100%
+ :alt: Breeze compile-ui-assets
+
Breeze cleanup
--------------
@@ -405,7 +419,7 @@ are several reasons why you might want to do that.
Breeze uses docker images heavily and those images are rebuild periodically and might leave dangling, unused
images in docker cache. This might cause extra disk usage. Also running various docker compose commands
-(for example running tests with ``breeze testing tests``) might create additional docker networks that might
+(for example running tests with ``breeze testing core-tests``) might create additional docker networks that might
prevent new networks from being created. Those networks are not removed automatically by docker-compose.
Also Breeze uses it's own cache to keep information about all images.
@@ -433,7 +447,7 @@ Then, next time when you start Breeze, it will have the data pre-populated.
These are all available flags of ``down`` command:
-.. image:: ./images/output-down.svg
+.. image:: ./images/output_down.svg
:target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_down.svg
:width: 100%
:alt: Breeze down
diff --git a/dev/breeze/doc/04_troubleshooting.rst b/dev/breeze/doc/04_troubleshooting.rst
index fd0b1dfa401..5e44c4dbf79 100644
--- a/dev/breeze/doc/04_troubleshooting.rst
+++ b/dev/breeze/doc/04_troubleshooting.rst
@@ -72,8 +72,51 @@ describe your problem.
stated in `This comment <https://github.com/moby/moby/issues/43361#issuecomment-1227617516>`_ and allows to
run Breeze with no problems.
-Bad Interpreter Error
----------------------
+Cannot import name 'cache' or Python >=3.9 required
+---------------------------------------------------
+
+When you see this error:
+
+.. code-block::
+
+ ImportError: cannot import name 'cache' from 'functools' (/Users/jarek/Library/Application Support/hatch/pythons/3.8/python/lib/python3.8/functools.py)
+
+or
+
+.. code-block::
+
+ ERROR: Package 'blacken-docs' requires a different Python: 3.8.18 not in '>=3.9'
+
+
+It means that your pre-commit hook is installed with (already End-Of-Life) Python 3.8 and you should reinstall
+it and clean pre-commit cache.
+
+This can be done with ``uv tool`` to install ``pre-commit``)
+
+.. code-block:: bash
+
+ uv tool uninstall pre-commit
+ uv tool install pre-commit --python 3.9 --force --with pre-commit-uv
+ pre-commit clean
+ pre-commit install
+
+You can also use ``pipx``
+
+.. code-block:: bash
+
+ pipx uninstall pre-commit
+ pipx install pre-commit --python $(which python3.9) --force
+ # This one allows pre-commit to use uv for venvs installed by pre-commit
+ pipx inject pre-commit pre-commit-uv # optionally if you want to use uv to install virtualenvs
+ pre-commit clean
+ pre-commit install
+
+If you installed ``pre-commit`` differently, you should remove and reinstall
+it (and clean cache) following the way you installed it.
+
+
+Bad Interpreter Error with ``pipx``
+-----------------------------------
If you are experiencing bad interpreter errors
``zsh: /Users/eladkal/.local/bin/breeze: bad interpreter: /Users/eladkal/.local/pipx/venvs/apache-airflow-breeze/bin/python: no such file or directory``
diff --git a/dev/breeze/doc/05_test_commands.rst b/dev/breeze/doc/05_test_commands.rst
index 79aa206921d..d9eb0856bbd 100644
--- a/dev/breeze/doc/05_test_commands.rst
+++ b/dev/breeze/doc/05_test_commands.rst
@@ -75,34 +75,28 @@ This applies to all kind of tests - all our tests can be run using pytest.
Running unit tests with ``breeze testing`` commands
...................................................
-An option you have is that you can also run tests via built-in ``breeze testing tests`` command - which
-is a "swiss-army-knife" of unit testing with Breeze. This command has a lot of parameters and is very
-flexible thus might be a bit overwhelming.
+An option you have is that you can also run tests via built-in ``breeze testing *tests*`` commands - which
+is a "swiss-army-knife" of unit testing with Breeze. You can run all groups of tests with that Airflow
+supports with one of the commands below.
-In most cases if you want to run tess you want to use dedicated ``breeze testing db-tests``
-or ``breeze testing non-db-tests`` commands that automatically run groups of tests that allow you to choose
-subset of tests to run (with ``--parallel-test-types`` flag)
+Using ``breeze testing core-tests`` command
+...........................................
-Using ``breeze testing tests`` command
-......................................
+The ``breeze testing core-tests`` command is that you can run for all or specify sub-set of the tests
+for Core.
-The ``breeze testing tests`` command is that you can easily specify sub-set of the tests -- including
-selecting specific Providers tests to run.
-
-For example this will only run provider tests for airbyte and http providers:
+For example this will run all core tests :
.. code-block:: bash
- breeze testing tests --test-type "Providers[airbyte,http]"
-
-You can also exclude tests for some providers from being run when whole "Providers" test type is run.
+ breeze testing core-tests
-For example this will run tests for all providers except amazon and google provider tests:
+For example this will only run "Other" tests :
.. code-block:: bash
- breeze testing tests --test-type "Providers[-amazon,google]"
+ breeze testing core-tests --test-type "Other"
You can also run parallel tests with ``--run-in-parallel`` flag - by default it will run all tests types
in parallel, but you can specify the test type that you want to run with space separated list of test
@@ -112,124 +106,161 @@ For example this will run API and WWW tests in parallel:
.. code-block:: bash
- breeze testing tests --parallel-test-types "API WWW" --run-in-parallel
+ breeze testing core-tests --parallel-test-types "API WWW" --run-in-parallel
-There are few special types of tests that you can run:
+Here is the detailed set of options for the ``breeze testing core-tests`` command.
-* ``All`` - all tests are run in single pytest run.
-* ``All-Postgres`` - runs all tests that require Postgres database
-* ``All-MySQL`` - runs all tests that require MySQL database
-* ``All-Quarantine`` - runs all tests that are in quarantine (marked with ``@pytest.mark.quarantined``
- decorator)
+.. image:: ./images/output_testing_core-tests.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_testing_core-tests.svg
+ :width: 100%
+ :alt: Breeze testing core-tests
-Here is the detailed set of options for the ``breeze testing tests`` command.
+Using ``breeze testing providers-tests`` command
+................................................
-.. image:: ./images/output_testing_tests.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_testing_tests.svg
- :width: 100%
- :alt: Breeze testing tests
+The ``breeze testing providers-tests`` command is that you can run for all or specify sub-set of the tests
+for Providers.
-Using ``breeze testing db-tests`` command
-.........................................
+For example this will run all provider tests tests :
-The ``breeze testing db-tests`` command is simplified version of the ``breeze testing tests`` command
-that only allows you to run tests that are not bound to a database - in parallel utilising all your CPUS.
-The DB-bound tests are the ones that require a database to be started and configured separately for
-each test type run and they are run in parallel containers/parallel docker compose projects to
-utilise multiple CPUs your machine has - thus allowing you to quickly run few groups of tests in parallel.
-This command is used in CI to run DB tests.
+.. code-block:: bash
-By default this command will run complete set of test types we have, thus allowing you to see result
-of all DB tests we have but you can choose a subset of test types to run by ``--parallel-test-types``
-flag or exclude some test types by specifying ``--excluded-parallel-test-types`` flag.
+ breeze testing providers-tests
-Run all DB tests:
+This will only run "amazon" and "google" provider tests :
.. code-block:: bash
- breeze testing db-tests
+ breeze testing providers-tests --test-type "Providers[amazon,google]"
-Only run DB tests from "API CLI WWW" test types:
+You can also run "all but" provider tests - this will run all providers tests except amazon and google :
.. code-block:: bash
- breeze testing db-tests --parallel-test-types "API CLI WWW"
+ breeze testing providers-tests --test-type "Providers[-amazon,google]"
+
+You can also run parallel tests with ``--run-in-parallel`` flag - by default it will run all tests types
+in parallel, but you can specify the test type that you want to run with space separated list of test
+types passed to ``--parallel-test-types`` flag.
-Run all DB tests excluding those in CLI and WWW test types:
+For example this will run ``amazon`` and ``google`` tests in parallel:
.. code-block:: bash
- breeze testing db-tests --excluded-parallel-test-types "CLI WWW"
+ breeze testing providers-tests --parallel-test-types "Providers[amazon] Providers[google]" --run-in-parallel
-Here is the detailed set of options for the ``breeze testing db-tests`` command.
+Here is the detailed set of options for the ``breeze testing providers-test`` command.
-.. image:: ./images/output_testing_db-tests.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_testing_db-tests.svg
+.. image:: ./images/output_testing_providers-tests.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_testing_providers-tests.svg
:width: 100%
- :alt: Breeze testing db-tests
+ :alt: Breeze testing providers-tests
+Using ``breeze testing task-sdk-tests`` command
+...............................................
-Using ``breeze testing non-db-tests`` command
-.........................................
+The ``breeze testing task-sdk-tests`` command is allows you to run tests for Task SDK without
+initializing database. The Task SDK should not need database to be started so this acts as a
+good check to see if the Task SDK tests are working properly.
-The ``breeze testing non-db-tests`` command is simplified version of the ``breeze testing tests`` command
-that only allows you to run tests that are not bound to a database - in parallel utilising all your CPUS.
-The non-DB-bound tests are the ones that do not expect a database to be started and configured and we can
-utilise multiple CPUs your machine has via ``pytest-xdist`` plugin - thus allowing you to quickly
-run few groups of tests in parallel using single container rather than many of them as it is the case for
-DB-bound tests. This command is used in CI to run Non-DB tests.
+Run all Task SDK tests:
-By default this command will run complete set of test types we have, thus allowing you to see result
-of all DB tests we have but you can choose a subset of test types to run by ``--parallel-test-types``
-flag or exclude some test types by specifying ``--excluded-parallel-test-types`` flag.
+.. code-block:: bash
-Run all non-DB tests:
+ breeze testing task-sdk-tests
-.. code-block:: bash
+Here is the detailed set of options for the ``breeze testing task-sdk-tests`` command.
- breeze testing non-db-tests
+.. image:: ./images/output_testing_task-sdk-tests.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_testing_task-sdk-tests.svg
+ :width: 100%
+ :alt: Breeze testing task-sdk-tests
-Only run non-DB tests from "API CLI WWW" test types:
-.. code-block:: bash
+Running integration core tests
+...............................
- breeze testing non-db-tests --parallel-test-types "API CLI WWW"
+You can also run integration core tests via built-in ``breeze testing core-integration-tests`` command.
+Some of our core tests require additional integrations to be started in docker-compose.
+The integration tests command will run the expected integration and tests that need that integration.
-Run all non-DB tests excluding those in CLI and WWW test types:
+For example this will only run kerberos tests:
.. code-block:: bash
- breeze testing non-db-tests --excluded-parallel-test-types "CLI WWW"
+ breeze testing core-integration-tests --integration kerberos
-Here is the detailed set of options for the ``breeze testing non-db-tests`` command.
+Here is the detailed set of options for the ``breeze testing core-integration-tests`` command.
-.. image:: ./images/output_testing_non-db-tests.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_testing_non-db-tests.svg
+.. image:: ./images/output_testing_core-integration-tests.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_testing_core-integration-tests.svg
:width: 100%
- :alt: Breeze testing non-db-tests
+ :alt: Breeze testing core-integration-tests
+Running integration providers tests
+...................................
-Running integration tests
-.........................
-
-You can also run integration tests via built-in ``breeze testing integration-tests`` command. Some of our
-tests require additional integrations to be started in docker-compose. The integration tests command will
-run the expected integration and tests that need that integration.
+You can also run integration core tests via built-in ``breeze testing providers-integration-tests`` command.
+Some of our core tests require additional integrations to be started in docker-compose.
+The integration tests command will run the expected integration and tests that need that integration.
For example this will only run kerberos tests:
.. code-block:: bash
- breeze testing integration-tests --integration kerberos
+ breeze testing providers-integration-tests --integration kerberos
+
+Here is the detailed set of options for the ``breeze testing providers-integration-tests`` command.
+
+.. image:: ./images/output_testing_providers-integration-tests.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_testing_providers-integration-tests.svg
+ :width: 100%
+ :alt: Breeze testing providers-integration-tests
+
+
+Running Python API client tests
+...............................
+
+To run Python API client tests, you need to have airflow python client packaged in dist folder.
+To package the client, clone the airflow-python-client repository and run the following command:
+
+.. code-block:: bash
+
+ breeze release-management prepare-python-client --package-format both
+ --version-suffix-for-pypi dev0 --python-client-repo ./airflow-client-python
+
+.. code-block:: bash
+ breeze testing python-api-client-tests
-Here is the detailed set of options for the ``breeze testing integration-tests`` command.
+Here is the detailed set of options for the ``breeze testing python-api-client-tests`` command.
-.. image:: ./images/output_testing_integration-tests.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_testing_integration_tests.svg
+.. image:: ./images/output_testing_python-api-client-tests.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_testing_python-api-client-tests.svg
:width: 100%
- :alt: Breeze testing integration-tests
+ :alt: Breeze testing python-api-client-tests
+
+Running system tests
+....................
+
+You can also run system core tests via built-in ``breeze testing system-tests`` command.
+Some of our core system tests runs against external systems and we can run them providing that
+credentials are configured to connect to those systems. Usually you should run only one or
+set of related tests this way.
+
+For example this will only run example_external_task_child_deferrable tests:
+
+.. code-block:: bash
+
+ breeze testing system-tests tests/system/example_empty.py
+
+Here is the detailed set of options for the ``breeze testing system-tests`` command.
+
+.. image:: ./images/output_testing_system-tests.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_testing_system-tests.svg
+ :width: 100%
+ :alt: Breeze testing system-tests
Running Helm unit tests
.......................
@@ -307,7 +338,7 @@ Kubernetes environment can be set with the ``breeze k8s setup-env`` command.
It will create appropriate virtualenv to run tests and download the right set of tools to run
the tests: ``kind``, ``kubectl`` and ``helm`` in the right versions. You can re-run the command
when you want to make sure the expected versions of the tools are installed properly in the
-virtualenv. The Virtualenv is available in ``.build/.k8s-env/bin`` subdirectory of your Airflow
+virtualenv. The Virtualenv is available in ``.build/k8s-env/bin`` subdirectory of your Airflow
installation.
.. image:: ./images/output_k8s_setup-env.svg
@@ -530,7 +561,7 @@ be created and airflow deployed to it before running the tests):
(kind-airflow-python-3.9-v1.24.0:KubernetesExecutor)> pytest test_kubernetes_executor.py
================================================= test session starts =================================================
- platform linux -- Python 3.10.6, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /home/jarek/code/airflow/.build/.k8s-env/bin/python
+ platform linux -- Python 3.10.6, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /home/jarek/code/airflow/.build/k8s-env/bin/python
cachedir: .pytest_cache
rootdir: /home/jarek/code/airflow, configfile: pytest.ini
plugins: anyio-3.6.1
@@ -540,8 +571,8 @@ be created and airflow deployed to it before running the tests):
test_kubernetes_executor.py::TestKubernetesExecutor::test_integration_run_dag_with_scheduler_failure PASSED [100%]
================================================== warnings summary ===================================================
- .build/.k8s-env/lib/python3.10/site-packages/_pytest/config/__init__.py:1233
- /home/jarek/code/airflow/.build/.k8s-env/lib/python3.10/site-packages/_pytest/config/__init__.py:1233: PytestConfigWarning: Unknown config option: asyncio_mode
+ .build/k8s-env/lib/python3.10/site-packages/_pytest/config/__init__.py:1233
+ /home/jarek/code/airflow/.build/k8s-env/lib/python3.10/site-packages/_pytest/config/__init__.py:1233: PytestConfigWarning: Unknown config option: asyncio_mode
self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")
diff --git a/dev/breeze/doc/06_managing_docker_images.rst b/dev/breeze/doc/06_managing_docker_images.rst
index 294f1540f36..ac0b0e6b1f6 100644
--- a/dev/breeze/doc/06_managing_docker_images.rst
+++ b/dev/breeze/doc/06_managing_docker_images.rst
@@ -76,7 +76,7 @@ These are all available flags of ``pull`` command:
Verifying CI image
..................
-Finally, you can verify CI image by running tests - either with the pulled/built images or
+You can verify CI image by running tests - either with the pulled/built images or
with an arbitrary image.
These are all available flags of ``verify`` command:
@@ -86,6 +86,86 @@ These are all available flags of ``verify`` command:
:width: 100%
:alt: Breeze ci-image verify
+Loading and saving CI image
+...........................
+
+You can load and save PROD image - for example to transfer it to another machine or to load an image
+that has been built in our CI.
+
+These are all available flags of ``save`` command:
+
+.. image:: ./images/output_ci-image_save.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_ci-image_save.svg
+ :width: 100%
+ :alt: Breeze ci-image save
+
+These are all available flags of ``load`` command:
+
+.. image:: ./images/output_ci-image_load.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_ci-image_load.svg
+ :width: 100%
+ :alt: Breeze ci-image load
+
+Images for every build from our CI are uploaded as artifacts to the
+GitHub Action run (in summary) and can be downloaded from there for 2 days in order to reproduce the complete
+environment used during the tests and loaded to the local Docker registry (note that you have
+to use the same platform as the CI run).
+
+You will find the artifacts for each image in the summary of the CI run. The artifacts are named
+``ci-image-docker-export-<platform>-<python_version>-<PR>_merge``. Those are compressed zip files that
+contain the ".tar" image that should be used with ``--image-file`` flag of the load method. Make sure to
+use the same ``--python`` version as the image was built with.
+
+To load the image from specific PR, you can use the following command:
+
+.. code-block:: bash
+
+ breeze ci-image load --from-pr 12345 --python 3.9 --github-token <your_github_token>
+
+To load the image from specific job run (for example 12538475388), you can use the following command, find the run id from github action runs.
+
+.. code-block:: bash
+
+ breeze ci-image load --from-run 12538475388 --python 3.9 --github-token <your_github_token>
+
+After you load the image, you can reproduce the very exact environment that was used in the CI run by
+entering breeze container without mounting your local sources:
+
+.. code-block:: bash
+
+ breeze shell --mount-sources skip [OTHER OPTIONS]
+
+And you should be able to run any tests and commands interactively in the very exact environment that
+was used in the failing CI run. This is a powerful tool to debug and fix CI issues.
+
+
+.. image:: ./images/image_artifacts.png
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_ci-image_load.svg
+ :width: 100%
+ :alt: Breeze image artifacts
+
+Exporting and importing CI image cache mount
+............................................
+
+During the build, cache of ``uv`` and ``pip`` is stored in a separate "cache mount" volum that is mounted
+during the build. This cache mount volume is preserved between builds and can be exported and imported
+to speed up the build process in CI - where cache is stored as artifact and can be imported in the next
+build.
+
+These are all available flags of ``export-mount-cache`` command:
+
+.. image:: ./images/output_ci-image_export-mount-cache.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_ci-image_export-mount-cache.svg
+ :width: 100%
+ :alt: Breeze ci-image
+
+These are all available flags of ``import-mount-cache`` command:
+
+.. image:: ./images/output_ci-image_import-mount-cache.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_ci-image_import-mount-cache.svg
+ :width: 100%
+ :alt: Breeze ci-image import-mount-cache
+
PROD Image tasks
----------------
@@ -140,10 +220,10 @@ suffix and they need to also be paired with corresponding runtime dependency add
.. code-block:: bash
- breeze prod-image build --python 3.8 --additional-dev-deps "libasound2-dev" \
+ breeze prod-image build --python 3.9 --additional-dev-deps "libasound2-dev" \
--additional-runtime-apt-deps "libasound2"
-Same as above but uses python 3.8.
+Same as above but uses python 3.9.
Building PROD image
...................
@@ -170,7 +250,7 @@ These are all available flags of ``pull-prod-image`` command:
Verifying PROD image
....................
-Finally, you can verify PROD image by running tests - either with the pulled/built images or
+You can verify PROD image by running tests - either with the pulled/built images or
with an arbitrary image.
These are all available flags of ``verify-prod-image`` command:
@@ -180,6 +260,31 @@ These are all available flags of ``verify-prod-image`` command:
:width: 100%
:alt: Breeze prod-image verify
+Loading and saving PROD image
+.............................
+
+You can load and save PROD image - for example to transfer it to another machine or to load an image
+that has been built in our CI.
+
+These are all available flags of ``save`` command:
+
+.. image:: ./images/output_prod-image_save.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_prod-image_save.svg
+ :width: 100%
+ :alt: Breeze prod-image save
+
+These are all available flags of ``load`` command:
+
+.. image:: ./images/output-prod-image_load.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_prod-image_load.svg
+ :width: 100%
+ :alt: Breeze prod-image load
+
+Similarly as in case of CI images, Images for every build from our CI are uploaded as artifacts to the
+GitHub Action run (in summary) and can be downloaded from there for 2 days in order to reproduce the complete
+environment used during the tests and loaded to the local Docker registry (note that you have
+to use the same platform as the CI run).
+
------
Next step: Follow the `Breeze maintenance tasks <07_breeze_maintenance_tasks.rst>`_ to learn about tasks that
diff --git a/dev/breeze/doc/09_release_management_tasks.rst b/dev/breeze/doc/09_release_management_tasks.rst
index 930f61159d1..9cef5d6ccd1 100644
--- a/dev/breeze/doc/09_release_management_tasks.rst
+++ b/dev/breeze/doc/09_release_management_tasks.rst
@@ -26,7 +26,7 @@ do not need or have no access to run). Those are usually connected with releasin
Those are all of the available release management commands:
.. image:: ./images/output_release-management.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_release-management.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management.svg
:width: 100%
:alt: Breeze release management
@@ -55,7 +55,7 @@ default is to build ``both`` type of packages ``sdist`` and ``wheel``.
breeze release-management prepare-airflow-package --package-format=wheel
.. image:: ./images/output_release-management_prepare-airflow-package.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_release-management_prepare-airflow-package.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_prepare-airflow-package.svg
:width: 100%
:alt: Breeze release-management prepare-airflow-package
@@ -79,7 +79,7 @@ tarball for.
breeze release-management prepare-airflow-tarball --version 2.8.0rc1
.. image:: ./images/output_release-management_prepare-airflow-tarball.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_release-management_prepare-airflow-tarball.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_prepare-airflow-tarball.svg
:width: 100%
:alt: Breeze release-management prepare-airflow-tarball
@@ -94,7 +94,7 @@ automates it.
breeze release-management create-minor-branch
.. image:: ./images/output_release-management_create-minor-branch.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_release-management_create-minor-branch.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_create-minor-branch.svg
:width: 100%
:alt: Breeze release-management create-minor-branch
@@ -109,7 +109,7 @@ When we prepare release candidate, we automate some of the steps we need to do.
breeze release-management start-rc-process
.. image:: ./images/output_release-management_start-rc-process.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_release-management_start-rc-process.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_start-rc-process.svg
:width: 100%
:alt: Breeze release-management start-rc-process
@@ -123,7 +123,7 @@ When we prepare final release, we automate some of the steps we need to do.
breeze release-management start-release
.. image:: ./images/output_release-management_start-release.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_release-management_start-rc-process.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_start-rc-process.svg
:width: 100%
:alt: Breeze release-management start-rc-process
@@ -154,7 +154,7 @@ You can also generate python client with custom security schemes.
These are all of the available flags for the command:
.. image:: ./images/output_release-management_prepare-python-client.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_release-management_prepare-python-client.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_prepare-python-client.svg
:width: 100%
:alt: Breeze release management prepare Python client
@@ -185,7 +185,7 @@ step can be skipped if you pass the ``--skip-latest`` flag.
These are all of the available flags for the ``release-prod-images`` command:
.. image:: ./images/output_release-management_release-prod-images.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_release-management_release-prod-images.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_release-prod-images.svg
:width: 100%
:alt: Breeze release management release prod images
@@ -208,7 +208,7 @@ However, If you want to disable this behaviour, set the envvar CLEAN_LOCAL_TAGS
These are all of the available flags for the ``tag-providers`` command:
.. image:: ./images/output_release-management_tag-providers.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_release-management_tag-providers.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_tag-providers.svg
:width: 100%
:alt: Breeze release management tag-providers
@@ -234,7 +234,7 @@ which version of Helm Chart you are preparing the tarball for.
breeze release-management prepare-helm-chart-tarball --version 1.12.0 --version-suffix rc1
.. image:: ./images/output_release-management_prepare-helm-chart-tarball.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_release-management_prepare-helm-chart-tarball.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_prepare-helm-chart-tarball.svg
:width: 100%
:alt: Breeze release-management prepare-helm-chart-tarball
@@ -256,7 +256,7 @@ This prepares helm chart .tar.gz package in the dist folder.
breeze release-management prepare-helm-chart-package --sign myemail@apache.org
.. image:: ./images/output_release-management_prepare-helm-chart-package.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_release-management_prepare-helm-chart-package.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_prepare-helm-chart-package.svg
:width: 100%
:alt: Breeze release-management prepare-helm-chart-package
@@ -292,7 +292,7 @@ The below example perform documentation preparation for provider packages.
You can also add ``--answer yes`` to perform non-interactive build.
.. image:: ./images/output_release-management_prepare-provider-documentation.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_release-management_prepare-provider-documentation.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_prepare-provider-documentation.svg
:width: 100%
:alt: Breeze prepare-provider-documentation
@@ -325,7 +325,7 @@ You can see all providers available by running this command:
breeze release-management prepare-provider-packages --help
.. image:: ./images/output_release-management_prepare-provider-packages.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_release-management_prepare-provider-packages.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_prepare-provider-packages.svg
:width: 100%
:alt: Breeze prepare-provider-packages
@@ -349,7 +349,7 @@ You can also run the verification with an earlier airflow version to check for c
All the command parameters are here:
.. image:: ./images/output_release-management_install-provider-packages.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_release-management_install-provider-packages.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_install-provider-packages.svg
:width: 100%
:alt: Breeze install-provider-packages
@@ -373,7 +373,7 @@ You can also run the verification with an earlier airflow version to check for c
All the command parameters are here:
.. image:: ./images/output_release-management_verify-provider-packages.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_release-management_verify-provider-packages.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_verify-provider-packages.svg
:width: 100%
:alt: Breeze verify-provider-packages
@@ -387,7 +387,7 @@ provider has been released) and date of the release of the provider version.
These are all of the available flags for the ``generate-providers-metadata`` command:
.. image:: ./images/output_release-management_generate-providers-metadata.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_release-management_generate-providers-metadata.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_generate-providers-metadata.svg
:width: 100%
:alt: Breeze release management generate providers metadata
@@ -398,7 +398,7 @@ Generating Provider Issue
You can use Breeze to generate a provider issue when you release new providers.
.. image:: ./images/output_release-management_generate-issue-content-providers.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_release-management_generate-issue-content-providers.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_generate-issue-content-providers.svg
:width: 100%
:alt: Breeze generate-issue-content-providers
@@ -414,7 +414,7 @@ command.
These are all available flags of ``clean-old-provider-artifacts`` command:
.. image:: ./images/output_release-management_clean-old-provider-artifacts.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_release-management_clean-old-provider-artifacts.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_clean-old-provider-artifacts.svg
:width: 100%
:alt: Breeze Clean Old Provider Artifacts
@@ -462,7 +462,7 @@ Constraints are generated separately for each python version and there are separ
These are all available flags of ``generate-constraints`` command:
.. image:: ./images/output_release-management_generate-constraints.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_release-management_generate-constraints.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_generate-constraints.svg
:width: 100%
:alt: Breeze generate-constraints
@@ -485,7 +485,7 @@ tagged already in the past. This can be done using ``breeze release-management u
These are all available flags of ``update-constraints`` command:
.. image:: ./images/output_release-management_update-constraints.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_release-management_update-constraints.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_update-constraints.svg
:width: 100%
:alt: Breeze update-constraints
@@ -552,7 +552,7 @@ publishing docs for multiple providers.
These are all available flags of ``release-management publish-docs`` command:
.. image:: ./images/output_release-management_publish-docs.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_release-management_publish-docs.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_publish-docs.svg
:width: 100%
:alt: Breeze Publish documentation
@@ -596,7 +596,7 @@ providers - you can mix apache-airflow, helm-chart and provider packages this wa
These are all available flags of ``release-management add-back-references`` command:
.. image:: ./images/output_release-management_add-back-references.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_release-management_add-back-references.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_add-back-references.svg
:width: 100%
:alt: Breeze Add Back References
@@ -606,7 +606,7 @@ SBOM generation tasks
Maintainers also can use Breeze for SBOM generation:
.. image:: ./images/output_sbom.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_sbom.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_sbom.svg
:width: 100%
:alt: Breeze sbom
@@ -619,7 +619,7 @@ done by the ``generate-providers-requirements`` command. This command generates
selected provider and python version, using the airflow version specified.
.. image:: ./images/output_sbom_generate-providers-requirements.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_sbom_generate-providers-requirements.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_sbom_generate-providers-requirements.svg
:width: 100%
:alt: Breeze generate SBOM provider requirements
@@ -634,7 +634,7 @@ information is written directly to ``docs-archive`` in airflow-site repository.
These are all of the available flags for the ``update-sbom-information`` command:
.. image:: ./images/output_sbom_update-sbom-information.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_sbomt_update-sbom-information.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_sbomt_update-sbom-information.svg
:width: 100%
:alt: Breeze update sbom information
@@ -646,7 +646,7 @@ such images are built with the ``build-all-airflow-images`` command.
This command will build one docker image per python version, with all the airflow versions >=2.0.0 compatible.
.. image:: ./images/output_sbom_build-all-airflow-images.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_sbom_build-all-airflow-images.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_sbom_build-all-airflow-images.svg
:width: 100%
:alt: Breeze build all airflow images
@@ -658,7 +658,7 @@ The SBOM information published on our website can be converted into a spreadshee
properties of the dependencies. This is done by the ``export-dependency-information`` command.
.. image:: ./images/output_sbom_export-dependency-information.svg
- :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_sbom_export-dependency-information.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_sbom_export-dependency-information.svg
:width: 100%
:alt: Breeze sbom export dependency information
@@ -666,3 +666,26 @@ properties of the dependencies. This is done by the ``export-dependency-informat
Next step: Follow the `Advanced Breeze topics <10_advanced_breeze_topics.rst>`_ to
learn more about Breeze internals.
+
+Preparing airflow Task SDK packages
+"""""""""""""""""""""""""""""""""""
+
+You can prepare airflow packages using Breeze:
+
+.. code-block:: bash
+
+ breeze release-management prepare-task-sdk-package
+
+This prepares airflow Task SDK .whl package in the dist folder.
+
+Again, you can specify optional ``--package-format`` flag to build selected formats of the Task SDK packages,
+default is to build ``both`` type of packages ``sdist`` and ``wheel``.
+
+.. code-block:: bash
+
+ breeze release-management prepare-task-sdk-package --package-format=wheel
+
+.. image:: ./images/output_release-management_prepare-task-sdk-package.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_prepare-airflow-package.svg
+ :width: 100%
+ :alt: Breeze release-management prepare-task-sdk-package
diff --git a/dev/breeze/doc/10_advanced_breeze_topics.rst b/dev/breeze/doc/10_advanced_breeze_topics.rst
index ac5421f85aa..9bbf113cb7e 100644
--- a/dev/breeze/doc/10_advanced_breeze_topics.rst
+++ b/dev/breeze/doc/10_advanced_breeze_topics.rst
@@ -29,17 +29,10 @@ Debugging/developing Breeze
Breeze can be quite easily debugged with PyCharm/VSCode or any other IDE - but it might be less discoverable
if you never tested modules and if you do not know how to bypass version check of breeze.
-For testing, you can create your own virtual environment, or use the one that ``pipx`` created for you if you
-already installed breeze following the recommended ``pipx install -e ./dev/breeze`` command.
+For testing, you can create your own virtual environment, or use the one that ``uv`` or ``pipx`` created
+for you if you already installed breeze following the recommended installation.
-For local virtualenv, you can use ``pyenv`` or any other virtualenv wrapper. For example with ``pyenv``,
-you can use ``pyenv virtualenv 3.8.6 airflow-breeze`` to create virtualenv called ``airflow-breeze``
-with Python 3.8.6. Then you can use ``pyenv activate airflow-breeze`` to activate it and install breeze
-in editable mode with ``pip install -e ./dev/breeze``.
-
-For ``pipx`` virtualenv, you can use the virtualenv that ``pipx`` created for you. You can find the name
-where ``pipx`` keeps their venvs via ``pipx list`` command. Usually it is
-``${HOME}/.local/pipx/venvs/apache-airflow-breeze`` where ``$HOME`` is your home directory.
+Or you can change your directory to
The venv can be used for running breeze tests and for debugging breeze. While running tests should
be usually "out-of-the-box" for most IDEs, once you configure ``./dev/breeze`` project to use the venv,
@@ -56,7 +49,7 @@ make sure to follow these steps:
this will bypass the check we run in Breeze to see if there are new requirements to install for it
See example configuration for PyCharm which has run/debug configuration for
-``breeze sbom generate-providers-requirements --provider-id sqlite --python 3.8``
+``breeze sbom generate-providers-requirements --provider-id sqlite --python 3.9``
.. raw:: html
diff --git a/dev/breeze/doc/adr/0002-implement-standalone-python-command.md b/dev/breeze/doc/adr/0002-implement-standalone-python-command.md
index 37eebcf3e15..ddd005fd92d 100644
--- a/dev/breeze/doc/adr/0002-implement-standalone-python-command.md
+++ b/dev/breeze/doc/adr/0002-implement-standalone-python-command.md
@@ -138,7 +138,7 @@ There are a few properties of Breeze/CI scripts that should be maintained though
run a command and get everything done with the least number of prerequisites
* The prerequisites for Breeze and CI are:
- * Python 3.8+ (Python 3.8 end of life is October 2024)
+ * Python 3.9+ (Python 3.9 end of life is October 2025)
* Docker (23.0+)
* Docker Compose (2.16.0+)
* No other tools and CLI commands should be needed
diff --git a/dev/breeze/doc/adr/0016-use-uv-tool-to-install-breeze.md b/dev/breeze/doc/adr/0016-use-uv-tool-to-install-breeze.md
new file mode 100644
index 00000000000..d425b6c40aa
--- /dev/null
+++ b/dev/breeze/doc/adr/0016-use-uv-tool-to-install-breeze.md
@@ -0,0 +1,56 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [10. Use uv tool to install breeze](#10-use-uv-tool-to-install-breeze)
+ - [Status](#status)
+ - [Context](#context)
+ - [Decision](#decision)
+ - [Consequences](#consequences)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# 10. Use uv tool to install breeze
+
+Date: 2024-11-11
+
+## Status
+
+Accepted
+
+Supersedes [10. Use pipx to install breeze](0010-use-pipx-to-install-breeze.md)
+
+## Context
+
+The ``uv`` tools is a new modern python development environment management tool
+and we adopt it in ``Airflow`` as recommended way to manage airflow local virtualenv and development
+setup. It's much faster to install dependencies with ``uv`` than with ``pip`` and it has many
+more features - including managing python interpreters, workspaces, syncing virtualenv and more.
+
+## Decision
+
+While it is still possible to install breeze using ``pipx``, we are now recommending ``uv`` and specifically
+``uv tool`` as the way to install breeze. Contributors should use ``uv tool`` to install breeze.
+
+## Consequences
+
+Those who used ``pipx``, should clean-up and reinstall their environment with ``uv``.
diff --git a/dev/breeze/doc/ci/01_ci_environment.md b/dev/breeze/doc/ci/01_ci_environment.md
index c9501a13b20..21044af5141 100644
--- a/dev/breeze/doc/ci/01_ci_environment.md
+++ b/dev/breeze/doc/ci/01_ci_environment.md
@@ -23,8 +23,9 @@
- [CI Environment](#ci-environment)
- [GitHub Actions workflows](#github-actions-workflows)
- - [Container Registry used as cache](#container-registry-used-as-cache)
+ - [GitHub Registry used as cache](#github-registry-used-as-cache)
- [Authentication in GitHub Registry](#authentication-in-github-registry)
+ - [GitHub Artifacts used to store built images](#github-artifacts-used-to-store-built-images)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
@@ -32,7 +33,8 @@
Continuous Integration is an important component of making Apache Airflow
robust and stable. We run a lot of tests for every pull request,
-for main and v2-\*-test branches and regularly as scheduled jobs.
+for `canary` runs from `main` and `v*-\*-test` branches
+regularly as scheduled jobs.
Our execution environment for CI is [GitHub Actions](https://github.com/features/actions). GitHub Actions.
@@ -60,57 +62,22 @@ To run the tests, we need to ensure that the images are built using the
latest sources and that the build process is efficient. A full rebuild
of such an image from scratch might take approximately 15 minutes.
Therefore, we've implemented optimization techniques that efficiently
-use the cache from the GitHub Docker registry. In most cases, this
-reduces the time needed to rebuild the image to about 4 minutes.
-However, when dependencies change, it can take around 6-7 minutes, and
-if the base image of Python releases a new patch-level, it can take
-approximately 12 minutes.
-
-## Container Registry used as cache
-
-We are using GitHub Container Registry to store the results of the
-`Build Images` workflow which is used in the `Tests` workflow.
-
-Currently in main version of Airflow we run tests in all versions of
-Python supported, which means that we have to build multiple images (one
-CI and one PROD for each Python version). Yet we run many jobs (\>15) -
-for each of the CI images. That is a lot of time to just build the
-environment to run. Therefore we are utilising the `pull_request_target`
-feature of GitHub Actions.
-
-This feature allows us to run a separate, independent workflow, when the
-main workflow is run -this separate workflow is different than the main
-one, because by default it runs using `main` version of the sources but
-also - and most of all - that it has WRITE access to the GitHub
-Container Image registry.
-
-This is especially important in our case where Pull Requests to Airflow
-might come from any repository, and it would be a huge security issue if
-anyone from outside could utilise the WRITE access to the Container
-Image Registry via external Pull Request.
-
-Thanks to the WRITE access and fact that the `pull_request_target` workflow named
-`Build Imaages` which - by default - uses the `main` version of the sources.
-There we can safely run some code there as it has been reviewed and merged.
-The workflow checks-out the incoming Pull Request, builds
-the container image from the sources from the incoming PR (which happens in an
-isolated Docker build step for security) and pushes such image to the
-GitHub Docker Registry - so that this image can be built only once and
-used by all the jobs running tests. The image is tagged with unique
-`COMMIT_SHA` of the incoming Pull Request and the tests run in the `pull` workflow
-can simply pull such image rather than build it from the scratch.
-Pulling such image takes ~ 1 minute, thanks to that we are saving a
-lot of precious time for jobs.
-
-We use [GitHub Container Registry](https://docs.github.com/en/packages/guides/about-github-container-registry).
-A `GITHUB_TOKEN` is needed to push to the registry. We configured
-scopes of the tokens in our jobs to be able to write to the registry,
-but only for the jobs that need it.
-
-The latest cache is kept as `:cache-linux-amd64` and `:cache-linux-arm64`
-tagged cache of our CI images (suitable for `--cache-from` directive of
-buildx). It contains metadata and cache for all segments in the image,
-and cache is kept separately for different platform.
+use the cache from Github Actions Artifacts.
+
+## GitHub Registry used as cache
+
+We are using GitHub Registry to store the last image built in canary run
+to build images in CI and local docker container.
+This is done to speed up the build process and to ensure that the
+first - time-consuming-to-build layers of the image are
+reused between the builds. The cache is stored in the GitHub Registry
+by the `canary` runs and then used in the subsequent runs.
+
+The latest GitHub registry cache is kept as `:cache-linux-amd64` and
+`:cache-linux-arm64` tagged cache of our CI images (suitable for
+`--cache-from` directive of buildx). It contains
+metadata and cache for all segments in the image,
+and cache is kept separately for different platforms.
The `latest` images of CI and PROD are `amd64` only images for CI,
because there is no easy way to push multiplatform images without
@@ -118,11 +85,25 @@ merging the manifests, and it is not really needed nor used for cache.
## Authentication in GitHub Registry
-We are using GitHub Container Registry as cache for our images.
-Authentication uses GITHUB_TOKEN mechanism. Authentication is needed for
-pushing the images (WRITE) only in `push`, `pull_request_target`
-workflows. When you are running the CI jobs in GitHub Actions,
-GITHUB_TOKEN is set automatically by the actions.
+Authentication to GitHub Registry in CI uses GITHUB_TOKEN mechanism.
+The Authentication is needed for pushing the images (WRITE) in the `canary` runs.
+When you are running the CI jobs in GitHub Actions, GITHUB_TOKEN is set automatically
+by the actions. This is used only in the `canary` runs that have "write" access
+to the repository.
+
+No `write` access is needed (nor possible) by Pull Requests coming from the forks,
+since we are only using "GitHub Artifacts" for cache source in those runs.
+
+## GitHub Artifacts used to store built images
+
+We are running most tests in reproducible CI image for all the jobs and
+instead of build the image multiple times we build image for each python
+version only once (one CI and one PROD). Those images are then used by
+All jobs that need them in the same build. The images - after building
+are exported to a file and stored in the GitHub Artifacts.
+The export files are then downloaded from artifacts and image is
+loaded from the file in all jobs in the same workflow after they are
+built and uploaded in the build image job.
----
diff --git a/dev/breeze/doc/ci/02_images.md b/dev/breeze/doc/ci/02_images.md
index 94d97c6962e..3d1d7d8b53e 100644
--- a/dev/breeze/doc/ci/02_images.md
+++ b/dev/breeze/doc/ci/02_images.md
@@ -43,9 +43,9 @@ Airflow has two main images (build from Dockerfiles):
production-ready Airflow installation. You can read more about
building and using the production image in the
[Docker stack](https://airflow.apache.org/docs/docker-stack/index.html)
- documentation. The image is built using [Dockerfile](Dockerfile).
+ documentation. The image is built using [Dockerfile](../../../../Dockerfile).
- CI image (Dockerfile.ci) - used for running tests and local
- development. The image is built using [Dockerfile.ci](Dockerfile.ci).
+ development. The image is built using [Dockerfile.ci](../../../../Dockerfile.ci).
## PROD image
@@ -108,7 +108,7 @@ it uses the latest installed version of airflow and providers. However,
you can choose different installation methods as described in [Building
PROD docker images from released PIP packages](#building-prod-docker-images-from-released-pip-packages). Detailed
reference for building production image from different sources can be
-found in: [Build Args reference](docs/docker-stack/build-arg-ref.rst#installing-airflow-using-different-methods)
+found in: [Build Args reference](../../../../docs/docker-stack/build-arg-ref.rst#installing-airflow-using-different-methods)
You can build the CI image using current sources this command:
@@ -126,20 +126,20 @@ By adding `--python <PYTHON_MAJOR_MINOR_VERSION>` parameter you can
build the image version for the chosen Python version.
The images are built with default extras - different extras for CI and
-production image and you can change the extras via the `--extras`
+production image and you can change the extras via the `--airflow-extras`
parameters and add new ones with `--additional-airflow-extras`.
-For example if you want to build Python 3.8 version of production image
+For example if you want to build Python 3.9 version of production image
with "all" extras installed you should run this command:
``` bash
-breeze prod-image build --python 3.8 --extras "all"
+breeze prod-image build --python 3.9 --airflow-extras "all"
```
If you just want to add new extras you can add them like that:
``` bash
-breeze prod-image build --python 3.8 --additional-airflow-extras "all"
+breeze prod-image build --python 3.9 --additional-airflow-extras "all"
```
The command that builds the CI image is optimized to minimize the time
@@ -160,7 +160,7 @@ You can also build production images from PIP packages via providing
`--install-airflow-version` parameter to Breeze:
``` bash
-breeze prod-image build --python 3.8 --additional-airflow-extras=trino --install-airflow-version=2.0.0
+breeze prod-image build --python 3.9 --additional-airflow-extras=trino --install-airflow-version=2.0.0
```
This will build the image using command similar to:
@@ -168,7 +168,7 @@ This will build the image using command similar to:
``` bash
pip install \
apache-airflow[async,amazon,celery,cncf.kubernetes,docker,elasticsearch,ftp,grpc,hashicorp,http,ldap,google,microsoft.azure,mysql,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv]==2.0.0 \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.0.0/constraints-3.8.txt"
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.0.0/constraints-3.9.txt"
```
> [!NOTE]
@@ -199,7 +199,7 @@ HEAD of development for constraints):
``` bash
pip install "https://github.com/apache/airflow/archive/<tag>.tar.gz#egg=apache-airflow" \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt"
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.9.txt"
```
You can also skip installing airflow and install it from locally
@@ -207,7 +207,7 @@ provided files by using `--install-packages-from-context` parameter to
Breeze:
``` bash
-breeze prod-image build --python 3.8 --additional-airflow-extras=trino --install-packages-from-context
+breeze prod-image build --python 3.9 --additional-airflow-extras=trino --install-packages-from-context
```
In this case you airflow and all packages (.whl files) should be placed
@@ -215,10 +215,11 @@ in `docker-context-files` folder.
# Using docker cache during builds
-Default mechanism used in Breeze for building CI images uses images
-pulled from GitHub Container Registry. This is done to speed up local
+Default mechanism used in Breeze for building CI images locally uses images
+pulled from GitHub Container Registry combined with locally mounted cache
+folders where `uv` cache is stored. This is done to speed up local
builds and building images for CI runs - instead of \> 12 minutes for
-rebuild of CI images, it takes usually about 1 minute when cache is
+rebuild of CI images, it takes usually less than a minute when cache is
used. For CI images this is usually the best strategy - to use default
"pull" cache. This is default strategy when [Breeze](../README.rst)
builds are performed.
@@ -227,7 +228,8 @@ For Production Image - which is far smaller and faster to build, it's
better to use local build cache (the standard mechanism that docker
uses. This is the default strategy for production images when
[Breeze](../README.rst) builds are
-performed. The first time you run it, it will take considerably longer
+performed. The local `uv` cache is used from mounted sources.
+The first time you run it, it will take considerably longer
time than if you use the pull mechanism, but then when you do small,
incremental changes to local sources, Dockerfile image and scripts,
further rebuilds with local build cache will be considerably faster.
@@ -241,20 +243,20 @@ flags: `registry` (default), `local`, or `disabled` flags when you run
Breeze commands. For example:
``` bash
-breeze ci-image build --python 3.8 --docker-cache local
+breeze ci-image build --python 3.9 --docker-cache local
```
Will build the CI image using local build cache (note that it will take
quite a long time the first time you run it).
``` bash
-breeze prod-image build --python 3.8 --docker-cache registry
+breeze prod-image build --python 3.9 --docker-cache registry
```
Will build the production image with cache used from registry.
``` bash
-breeze prod-image build --python 3.8 --docker-cache disabled
+breeze prod-image build --python 3.9 --docker-cache disabled
```
Will build the production image from the scratch.
@@ -293,19 +295,12 @@ See
Naming convention for the GitHub packages.
-Images with a commit SHA (built for pull requests and pushes). Those are
-images that are snapshot of the currently run build. They are built once
-per each build and pulled by each test job.
-
``` bash
-ghcr.io/apache/airflow/<BRANCH>/ci/python<X.Y>:<COMMIT_SHA> - for CI images
-ghcr.io/apache/airflow/<BRANCH>/prod/python<X.Y>:<COMMIT_SHA> - for production images
+ghcr.io/apache/airflow/<BRANCH>/ci/python<X.Y> - for CI images
+ghcr.io/apache/airflow/<BRANCH>/prod/python<X.Y> - for production images
```
-Thoe image contain inlined cache.
-
-You can see all the current GitHub images at
-<https://github.com/apache/airflow/packages>
+You can see all the current GitHub images at <https://github.com/apache/airflow/packages>
Note that you need to be committer and have the right to refresh the
images in the GitHub Registry with latest sources from main via
@@ -314,12 +309,23 @@ need to login with your Personal Access Token with "packages" write
scope to be able to push to those repositories or pull from them in case
of GitHub Packages.
-GitHub Container Registry
+You need to login to GitHub Container Registry with your API token
+if you want to interact with the GitHub Registry for writing (only
+committers).
``` bash
docker login ghcr.io
```
+Note that when your token is expired and you are still
+logged in, you are not able to interact even with read-only operations
+like pulling images. You need to logout and login again to refresh the
+token.
+
+``` bash
+docker logout ghcr.io
+```
+
Since there are different naming conventions used for Airflow images and
there are multiple images used, [Breeze](../README.rst)
provides easy to use management interface for the images. The CI
@@ -329,22 +335,14 @@ new version of base Python is released. However, occasionally, you might
need to rebuild images locally and push them directly to the registries
to refresh them.
-Every developer can also pull and run images being result of a specific
+Every contributor can also pull and run images being result of a specific
CI run in GitHub Actions. This is a powerful tool that allows to
reproduce CI failures locally, enter the images and fix them much
-faster. It is enough to pass `--image-tag` and the registry and Breeze
-will download and execute commands using the same image that was used
-during the CI tests.
+faster. It is enough to download and uncompress the artifact that stores the
+image and run ``breeze ci-image load -i <path-to-image.tar>`` to load the
+image and mark the image as refreshed in the local cache.
-For example this command will run the same Python 3.8 image as was used
-in build identified with 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e commit
-SHA with enabled rabbitmq integration.
-
-``` bash
-breeze --image-tag 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e --python 3.8 --integration rabbitmq
-```
-
-You can see more details and examples in[Breeze](../README.rst)
+You can see more details and examples in[Breeze](../06_managing_docker_images.rst)
# Customizing the CI image
@@ -361,7 +359,7 @@ you can build the image in the
Here just a few examples are presented which should give you general
understanding of what you can customize.
-This builds the production image in version 3.8 with additional airflow
+This builds the production image in version 3.9 with additional airflow
extras from 2.0.0 PyPI package and additional apt dev and runtime
dependencies.
@@ -373,7 +371,7 @@ plugin installed.
``` bash
DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \
--pull \
- --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-bookworm" \
+ --build-arg PYTHON_BASE_IMAGE="python:3.9-slim-bookworm" \
--build-arg ADDITIONAL_AIRFLOW_EXTRAS="jdbc" \
--build-arg ADDITIONAL_PYTHON_DEPS="pandas" \
--build-arg ADDITIONAL_DEV_APT_DEPS="gcc g++" \
@@ -384,7 +382,7 @@ the same image can be built using `breeze` (it supports auto-completion
of the options):
``` bash
-breeze ci-image build --python 3.8 --additional-airflow-extras=jdbc --additional-python-deps="pandas" \
+breeze ci-image build --python 3.9 --additional-airflow-extras=jdbc --additional-python-deps="pandas" \
--additional-dev-apt-deps="gcc g++"
```
@@ -398,7 +396,7 @@ comment](https://github.com/apache/airflow/issues/8605#issuecomment-690065621):
``` bash
DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \
--pull \
- --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-bookworm" \
+ --build-arg PYTHON_BASE_IMAGE="python:3.9-slim-bookworm" \
--build-arg AIRFLOW_INSTALLATION_METHOD="apache-airflow" \
--build-arg ADDITIONAL_AIRFLOW_EXTRAS="slack" \
--build-arg ADDITIONAL_PYTHON_DEPS="apache-airflow-providers-odbc \
@@ -421,96 +419,93 @@ DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \
The following build arguments (`--build-arg` in docker build command)
can be used for CI images:
-| Build argument | Default value | Description |
-|-----------------------------------|----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `PYTHON_BASE_IMAGE` | `python:3.8-slim-bookworm` | Base Python image |
-| `PYTHON_MAJOR_MINOR_VERSION` | `3.8` | major/minor version of Python (should match base image) |
-| `DEPENDENCIES_EPOCH_NUMBER` | `2` | increasing this number will reinstall all apt dependencies |
-| `ADDITIONAL_PIP_INSTALL_FLAGS` | | additional `pip` flags passed to the installation commands (except when reinstalling `pip` itself) |
-| `PIP_NO_CACHE_DIR` | `true` | if true, then no pip cache will be stored |
-| `UV_NO_CACHE` | `true` | if true, then no uv cache will be stored |
-| `HOME` | `/root` | Home directory of the root user (CI image has root user as default) |
-| `AIRFLOW_HOME` | `/root/airflow` | Airflow's HOME (that's where logs and sqlite databases are stored) |
-| `AIRFLOW_SOURCES` | `/opt/airflow` | Mounted sources of Airflow |
-| `AIRFLOW_REPO` | `apache/airflow` | the repository from which PIP dependencies are pre-installed |
-| `AIRFLOW_BRANCH` | `main` | the branch from which PIP dependencies are pre-installed |
-| `AIRFLOW_CI_BUILD_EPOCH` | `1` | increasing this value will reinstall PIP dependencies from the repository from scratch |
-| `AIRFLOW_CONSTRAINTS_LOCATION` | | If not empty, it will override the source of the constraints with the specified URL or file. |
-| `AIRFLOW_CONSTRAINTS_REFERENCE` | | reference (branch or tag) from GitHub repository from which constraints are used. By default it is set to `constraints-main` but can be `constraints-2-X`. |
-| `AIRFLOW_EXTRAS` | `all` | extras to install |
-| `UPGRADE_INVALIDATION_STRING` | | If set to any random value the dependencies are upgraded to newer versions. In CI it is set to build id. |
-| `AIRFLOW_PRE_CACHED_PIP_PACKAGES` | `true` | Allows to pre-cache airflow PIP packages from the GitHub of Apache Airflow This allows to optimize iterations for Image builds and speeds up CI jobs. |
-| `ADDITIONAL_AIRFLOW_EXTRAS` | | additional extras to install |
-| `ADDITIONAL_PYTHON_DEPS` | | additional Python dependencies to install |
-| `DEV_APT_COMMAND` | | Dev apt command executed before dev deps are installed in the first part of image |
-| `ADDITIONAL_DEV_APT_COMMAND` | | Additional Dev apt command executed before dev dep are installed in the first part of the image |
-| `DEV_APT_DEPS` | | Dev APT dependencies installed in the first part of the image |
-| `ADDITIONAL_DEV_APT_DEPS` | | Additional apt dev dependencies installed in the first part of the image |
-| `ADDITIONAL_DEV_APT_ENV` | | Additional env variables defined when installing dev deps |
-| `AIRFLOW_PIP_VERSION` | `24.3.1` | PIP version used. |
-| `AIRFLOW_UV_VERSION` | `0.5.11` | UV version used. |
-| `AIRFLOW_USE_UV` | `true` | Whether to use UV for installation. |
-| `PIP_PROGRESS_BAR` | `on` | Progress bar for PIP installation |
-
-
-The"
+| Build argument | Default value | Description |
+|---------------------------------|----------------------------|-------------------------------------------------------------------------------------------------------------------|
+| `PYTHON_BASE_IMAGE` | `python:3.9-slim-bookworm` | Base Python image |
+| `PYTHON_MAJOR_MINOR_VERSION` | `3.9` | major/minor version of Python (should match base image) |
+| `DEPENDENCIES_EPOCH_NUMBER` | `2` | increasing this number will reinstall all apt dependencies |
+| `ADDITIONAL_PIP_INSTALL_FLAGS` | | additional `pip` flags passed to the installation commands (except when reinstalling `pip` itself) |
+| `HOME` | `/root` | Home directory of the root user (CI image has root user as default) |
+| `AIRFLOW_HOME` | `/root/airflow` | Airflow's HOME (that's where logs and sqlite databases are stored) |
+| `AIRFLOW_SOURCES` | `/opt/airflow` | Mounted sources of Airflow |
+| `AIRFLOW_REPO` | `apache/airflow` | the repository from which PIP dependencies are pre-installed |
+| `AIRFLOW_BRANCH` | `main` | the branch from which PIP dependencies are pre-installed |
+| `AIRFLOW_CI_BUILD_EPOCH` | `1` | increasing this value will reinstall PIP dependencies from the repository from scratch |
+| `AIRFLOW_CONSTRAINTS_LOCATION` | | If not empty, it will override the source of the constraints with the specified URL or file. |
+| `AIRFLOW_CONSTRAINTS_REFERENCE` | `constraints-main` | reference (branch or tag) from GitHub repository from which constraints are used. |
+| `AIRFLOW_EXTRAS` | `all` | extras to install |
+| `UPGRADE_INVALIDATION_STRING` | | If set to any random value the dependencies are upgraded to newer versions. In CI it is set to build id. |
+| `ADDITIONAL_AIRFLOW_EXTRAS` | | additional extras to install |
+| `ADDITIONAL_PYTHON_DEPS` | | additional Python dependencies to install |
+| `DEV_APT_COMMAND` | | Dev apt command executed before dev deps are installed in the first part of image |
+| `ADDITIONAL_DEV_APT_COMMAND` | | Additional Dev apt command executed before dev dep are installed in the first part of the image |
+| `DEV_APT_DEPS` | | Dev APT dependencies installed in the first part of the image (default empty means default dependencies are used) |
+| `ADDITIONAL_DEV_APT_DEPS` | | Additional apt dev dependencies installed in the first part of the image |
+| `ADDITIONAL_DEV_APT_ENV` | | Additional env variables defined when installing dev deps |
+| `AIRFLOW_PIP_VERSION` | `24.3.1` | `pip` version used. |
+| `AIRFLOW_UV_VERSION` | `0.5.14` | `uv` version used. |
+| `AIRFLOW_PRE_COMMIT_VERSION` | `4.0.1` | `pre-commit` version used. |
+| `AIRFLOW_PRE_COMMIT_UV_VERSION` | `4.1.4` | `pre-commit-uv` version used. |
+| `AIRFLOW_USE_UV` | `true` | Whether to use UV for installation. |
+| `PIP_PROGRESS_BAR` | `on` | Progress bar for PIP installation |
+
Here are some examples of how CI images can built manually. CI is always
built from local sources.
-This builds the CI image in version 3.8 with default extras ("all").
+This builds the CI image in version 3.9 with default extras ("all").
``` bash
DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \
--pull \
- --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-bookworm" --tag my-image:0.0.1
+ --build-arg PYTHON_BASE_IMAGE="python:3.9-slim-bookworm" --tag my-image:0.0.1
```
-This builds the CI image in version 3.8 with "gcp" extra only.
+This builds the CI image in version 3.9 with "gcp" extra only.
``` bash
DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \
--pull \
- --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-bookworm" \
+ --build-arg PYTHON_BASE_IMAGE="python:3.9-slim-bookworm" \
--build-arg AIRFLOW_EXTRAS=gcp --tag my-image:0.0.1
```
-This builds the CI image in version 3.8 with "apache-beam" extra added.
+This builds the CI image in version 3.9 with "apache-beam" extra added.
``` bash
DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \
--pull \
- --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-bookworm" \
+ --build-arg PYTHON_BASE_IMAGE="python:3.9-slim-bookworm" \
--build-arg ADDITIONAL_AIRFLOW_EXTRAS="apache-beam" --tag my-image:0.0.1
```
-This builds the CI image in version 3.8 with "mssql" additional package
+This builds the CI image in version 3.9 with "mssql" additional package
added.
``` bash
DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \
--pull \
- --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-bookworm" \
+ --build-arg PYTHON_BASE_IMAGE="python:3.9-slim-bookworm" \
--build-arg ADDITIONAL_PYTHON_DEPS="mssql" --tag my-image:0.0.1
```
-This builds the CI image in version 3.8 with "gcc" and "g++" additional
+This builds the CI image in version 3.9 with "gcc" and "g++" additional
apt dev dependencies added.
```
DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \
--pull
- --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-bookworm" \
+ --build-arg PYTHON_BASE_IMAGE="python:3.9-slim-bookworm" \
--build-arg ADDITIONAL_DEV_APT_DEPS="gcc g++" --tag my-image:0.0.1
```
-This builds the CI image in version 3.8 with "jdbc" extra and
+This builds the CI image in version 3.9 with "jdbc" extra and
"default-jre-headless" additional apt runtime dependencies added.
```
DOCKER_BUILDKIT=1 docker build . -f Dockerfile.ci \
--pull \
- --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-bookworm" \
+ --build-arg PYTHON_BASE_IMAGE="python:3.9-slim-bookworm" \
--build-arg AIRFLOW_EXTRAS=jdbc \
--tag my-image:0.0.1
```
@@ -545,8 +540,8 @@ The entrypoint performs those operations:
sets the right pytest flags
- Sets default "tests" target in case the target is not explicitly set
as additional argument
-- Runs system tests if RUN_SYSTEM_TESTS flag is specified, otherwise
- runs regular unit and integration tests
+- Runs system tests if TEST_GROUP is "system-core" or "system-providers"
+ otherwise runs regular unit and integration tests
# Naming conventions for stored images
@@ -555,10 +550,6 @@ The images produced during the `Build Images` workflow of CI jobs are
stored in the [GitHub Container
Registry](https://github.com/orgs/apache/packages?repo_name=airflow)
-The images are stored with both "latest" tag (for last main push image
-that passes all the tests as well with the COMMIT_SHA id for images that
-were used in particular build.
-
The image names follow the patterns (except the Python image, all the
images are stored in <https://ghcr.io/> in `apache` organization.
@@ -569,22 +560,15 @@ percent-encoded when you access them via UI (/ = %2F)
`https://github.com/apache/airflow/pkgs/container/<CONTAINER_NAME>`
-| Image | Name:tag (both cases latest version and per-build) | Description |
-|--------------------------|----------------------------------------------------|---------------------------------------------------------------|
-| Python image (DockerHub) | python:\<X.Y\>-slim-bookworm | Base Python image used by both production and CI image. |
-| CI image | airflow/\<BRANCH\>/ci/python\<X.Y\>:\<TAG\> | CI image - this is the image used for most of the tests. |
-| PROD image | airflow/\<BRANCH\>/prod/python\<X.Y\>:\<TAG\> | faster to build or pull. Production image optimized for size. |
+| Image | Name | Description |
+|--------------------------|----------------------------------------|---------------------------------------------------------------|
+| Python image (DockerHub) | python:\<X.Y\>-slim-bookworm | Base Python image used by both production and CI image. |
+| CI image | airflow/\<BRANCH\>/ci/python\<X.Y\> | CI image - this is the image used for most of the tests. |
+| PROD image | airflow/\<BRANCH\>/prod/python\<X.Y\> | faster to build or pull. Production image optimized for size. |
- \<BRANCH\> might be either "main" or "v2-\*-test"
-- \<X.Y\> - Python version (Major + Minor).Should be one of \["3.8",
- "3.9", "3.10", "3.11", "3.12" \].
-- \<COMMIT_SHA\> - full-length SHA of commit either from the tip of the
- branch (for pushes/schedule) or commit from the tip of the branch used
- for the PR.
-- \<TAG\> - tag of the image. It is either "latest" or \<COMMIT_SHA\>
- (full-length SHA of commit either from the tip of the branch (for
- pushes/schedule) or commit from the tip of the branch used for the
- PR).
+- \<X.Y\> - Python version (Major + Minor).Should be one of \["3.9", "3.10", "3.11", "3.12" \].
+
----
diff --git a/dev/breeze/doc/ci/03_github_variables.md b/dev/breeze/doc/ci/03_github_variables.md
index 10983369784..bf1353df080 100644
--- a/dev/breeze/doc/ci/03_github_variables.md
+++ b/dev/breeze/doc/ci/03_github_variables.md
@@ -71,4 +71,4 @@ docker tag ghcr.io/apache/airflow/main/ci/python3.10 your-image-name:tag
-----
-Read next about [Static checks](04_static_checks.md)
+Read next about [Selective checks](04_selective_checks.md)
diff --git a/dev/breeze/doc/ci/04_selective_checks.md b/dev/breeze/doc/ci/04_selective_checks.md
index 3f8d8a97fae..80942370843 100644
--- a/dev/breeze/doc/ci/04_selective_checks.md
+++ b/dev/breeze/doc/ci/04_selective_checks.md
@@ -27,7 +27,7 @@
- [Skipping pre-commits (Static checks)](#skipping-pre-commits-static-checks)
- [Suspended providers](#suspended-providers)
- [Selective check outputs](#selective-check-outputs)
- - [Committer vs. non-committer PRs](#committer-vs-non-committer-prs)
+ - [Committer vs. Non-committer PRs](#committer-vs-non-committer-prs)
- [Changing behaviours of the CI runs by setting labels](#changing-behaviours-of-the-ci-runs-by-setting-labels)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
@@ -59,6 +59,7 @@ We have the following Groups of files for CI that determine which tests are run:
provider and `hatch_build.py` for all regular dependencies.
* `DOC files` - change in those files indicate that we should run documentation builds (both airflow sources
and airflow documentation)
+* `UI files` - those are files for the new full React UI (useful to determine if UI tests should run)
* `WWW files` - those are files for the WWW part of our UI (useful to determine if UI tests should run)
* `System test files` - those are the files that are part of system tests (system tests are not automatically
run in our CI, but Airflow stakeholders are running the tests and expose dashboards for them at
@@ -143,6 +144,7 @@ when some files are not changed. Those are the rules implemented:
* if no `All Airflow Python files` changed - `mypy-airflow` check is skipped
* if no `All Docs Python files` changed - `mypy-docs` check is skipped
* if no `All Dev Python files` changed - `mypy-dev` check is skipped
+ * if no `UI files` changed - `ts-compile-format-lint-ui` check is skipped
* if no `WWW files` changed - `ts-compile-format-lint-www` check is skipped
* if no `All Python files` changed - `flynt` check is skipped
* if no `Helm files` changed - `lint-helm-chart` check is skipped
@@ -164,73 +166,90 @@ separated by spaces. This is to accommodate for the wau how outputs of this kind
Github Actions to pass the list of parameters to a command to execute
-| Output | Meaning of the output | Example value | List as string |
-|----------------------------------------|------------------------------------------------------------------------------------------------------|-------------------------------------------|----------------|
-| affected-providers-list-as-string | List of providers affected when they are selectively affected. | airbyte http | * |
-| all-python-versions | List of all python versions there are available in the form of JSON array | ['3.8', '3.9', '3.10'] | |
-| all-python-versions-list-as-string | List of all python versions there are available in the form of space separated string | 3.8 3.9 3.10 | * |
-| all-versions | If set to true, then all python, k8s, DB versions are used for tests. | false | |
-| basic-checks-only | Whether to run all static checks ("false") or only basic set of static checks ("true") | false | |
-| build_system_changed_in_pyproject_toml | When builds system dependencies changed in pyproject.toml changed in the PR. | false | |
-| chicken-egg-providers | List of providers that should be considered as "chicken-egg" - expecting development Airflow version | | |
-| ci-image-build | Whether CI image build is needed | true | |
-| debug-resources | Whether resources usage should be printed during parallel job execution ("true"/ "false") | false | |
-| default-branch | Which branch is default for the build ("main" for main branch, "v2-4-test" for 2.4 line etc.) | main | |
-| default-constraints-branch | Which branch is default for the build ("constraints-main" for main branch, "constraints-2-4" etc.) | constraints-main | |
-| default-helm-version | Which Helm version to use as default | v3.9.4 | |
-| default-kind-version | Which Kind version to use as default | v0.16.0 | |
-| default-kubernetes-version | Which Kubernetes version to use as default | v1.25.2 | |
-| default-mysql-version | Which MySQL version to use as default | 5.7 | |
-| default-postgres-version | Which Postgres version to use as default | 10 | |
-| default-python-version | Which Python version to use as default | 3.8 | |
-| docker-cache | Which cache should be used for images ("registry", "local" , "disabled") | registry | |
-| docs-build | Whether to build documentation ("true"/"false") | true | |
-| docs-list-as-string | What filter to apply to docs building - based on which documentation packages should be built | apache-airflow helm-chart google | |
-| full-tests-needed | Whether this build runs complete set of tests or only subset (for faster PR builds) [1] | false | |
-| generated-dependencies-changed | Whether generated dependencies have changed ("true"/"false") | false | |
-| hatch-build-changed | When hatch build.py changed in the PR. | false | |
-| helm-version | Which Helm version to use for tests | v3.9.4 | |
-| is-airflow-runner | Whether runner used is an airflow or infrastructure runner (true if airflow/false if infrastructure) | false | |
-| is-amd-runner | Whether runner used is an AMD one | true | |
-| is-arm-runner | Whether runner used is an ARM one | false | |
-| is-committer-build | Whether the build is triggered by a committer | false | |
-| is-k8s-runner | Whether the build runs on our k8s infrastructure | false | |
-| is-self-hosted-runner | Whether the runner is self-hosted | false | |
-| is-vm-runner | Whether the runner uses VM to run | true | |
-| kind-version | Which Kind version to use for tests | v0.16.0 | |
-| kubernetes-combos-list-as-string | All combinations of Python version and Kubernetes version to use for tests as space-separated string | 3.8-v1.25.2 3.9-v1.26.4 | * |
-| kubernetes-versions | All Kubernetes versions to use for tests as JSON array | ['v1.25.2'] | |
-| kubernetes-versions-list-as-string | All Kubernetes versions to use for tests as space-separated string | v1.25.2 | * |
-| mypy-checks | List of folders to be considered for mypy | [] | |
-| mysql-exclude | Which versions of MySQL to exclude for tests as JSON array | [] | |
-| mysql-versions | Which versions of MySQL to use for tests as JSON array | ['5.7'] | |
-| needs-api-codegen | Whether "api-codegen" are needed to run ("true"/"false") | true | |
-| needs-api-tests | Whether "api-tests" are needed to run ("true"/"false") | true | |
-| needs-helm-tests | Whether Helm tests are needed to run ("true"/"false") | true | |
-| needs-javascript-scans | Whether javascript CodeQL scans should be run ("true"/"false") | true | |
-| needs-mypy | Whether mypy check is supposed to run in this build | true | |
-| needs-python-scans | Whether Python CodeQL scans should be run ("true"/"false") | true | |
-| parallel-test-types-list-as-string | Which test types should be run for unit tests | API Always Providers Providers\[-google\] | * |
-| postgres-exclude | Which versions of Postgres to exclude for tests as JSON array | [] | |
-| postgres-versions | Which versions of Postgres to use for tests as JSON array | ['10'] | |
-| prod-image-build | Whether PROD image build is needed | true | |
-| prod-image-build | Whether PROD image build is needed | true | |
-| providers-compatibility-checks | List of dicts: (python_version, airflow_version, removed_providers) for compatibility checks | [] | |
-| pyproject-toml-changed | When pyproject.toml changed in the PR. | false | |
-| python-versions | List of python versions to use for that build | ['3.8'] | * |
-| python-versions-list-as-string | Which versions of MySQL to use for tests as space-separated string | 3.8 | * |
-| run-amazon-tests | Whether Amazon tests should be run ("true"/"false") | true | |
-| run-kubernetes-tests | Whether Kubernetes tests should be run ("true"/"false") | true | |
-| run-tests | Whether unit tests should be run ("true"/"false") | true | |
-| run-www-tests | Whether WWW tests should be run ("true"/"false") | true | |
-| runs-on-as-json-default | List of labels assigned for runners for that build for default runs for that build (as string) | ["ubuntu-22.04"] | |
-| runs-on-as-json-self-hosted | List of labels assigned for runners for that build for self hosted runners | ["self-hosted", "Linux", "X64"] | |
-| runs-on-as-json-public | List of labels assigned for runners for that build for public runners | ["ubuntu-22.04"] | |
-| skip-pre-commits | Which pre-commits should be skipped during the static-checks run | check-provider-yaml-valid,flynt,identity | |
-| skip-provider-tests | When provider tests should be skipped (on non-main branch or when no provider changes detected) | true | |
-| sqlite-exclude | Which versions of Sqlite to exclude for tests as JSON array | [] | |
-| testable-integrations | List of integrations that are testable in the build as JSON array | ['mongo', 'kafka', 'mssql'] | |
-| upgrade-to-newer-dependencies | Whether the image build should attempt to upgrade all dependencies (true/false or commit hash) | false | |
+| Output | Meaning of the output | Example value | List |
+|------------------------------------------------|--------------------------------------------------------------------------------------------------------|-----------------------------------------|------|
+| all-python-versions | List of all python versions there are available in the form of JSON array | \['3.9', '3.10'\] | |
+| all-python-versions-list-as-string | List of all python versions there are available in the form of space separated string | 3.9 3.10 | * |
+| all-versions | If set to true, then all python, k8s, DB versions are used for tests. | false | |
+| basic-checks-only | Whether to run all static checks ("false") or only basic set of static checks ("true") | false | |
+| build_system_changed_in_pyproject_toml | When builds system dependencies changed in pyproject.toml changed in the PR. | false | |
+| chicken-egg-providers | List of providers that should be considered as "chicken-egg" - expecting development Airflow version | | |
+| ci-image-build | Whether CI image build is needed | true | |
+| core-test-types-list-as-string | Which test types should be run for unit tests for core | API Always Providers | * |
+| debug-resources | Whether resources usage should be printed during parallel job execution ("true"/ "false") | false | |
+| default-branch | Which branch is default for the build ("main" for main branch, "v2-4-test" for 2.4 line etc.) | main | |
+| default-constraints-branch | Which branch is default for the build ("constraints-main" for main branch, "constraints-2-4" etc.) | constraints-main | |
+| default-helm-version | Which Helm version to use as default | v3.9.4 | |
+| default-kind-version | Which Kind version to use as default | v0.16.0 | |
+| default-kubernetes-version | Which Kubernetes version to use as default | v1.25.2 | |
+| default-mysql-version | Which MySQL version to use as default | 5.7 | |
+| default-postgres-version | Which Postgres version to use as default | 10 | |
+| default-python-version | Which Python version to use as default | 3.9 | |
+| disable-airflow-repo-cache | Disables cache of the repo main cache in CI - aiflow will be installed without main installation cache | true | |
+| docker-cache | Which cache should be used for images ("registry", "local" , "disabled") | registry | |
+| docs-build | Whether to build documentation ("true"/"false") | true | |
+| docs-list-as-string | What filter to apply to docs building - based on which documentation packages should be built | apache-airflow helm-chart google | * |
+| excluded-providers-as-string c | List of providers that should be excluded from the build as space-separated string | amazon google | * |
+| force-pip | Whether pip should be forced in the image build instead of uv ("true"/"false") | false | |
+| full-tests-needed | Whether this build runs complete set of tests or only subset (for faster PR builds) \[1\] | false | |
+| generated-dependencies-changed | Whether generated dependencies have changed ("true"/"false") | false | |
+| has-migrations | Whether the PR has migrations ("true"/"false") | false | |
+| hatch-build-changed | When hatch build.py changed in the PR. | false | |
+| helm-test-packages-list-as-string | List of helm packages to test as JSON array | \["airflow_aux", "airflow_core"\] | * |
+| helm-version | Which Helm version to use for tests | v3.15.3 | |
+| include-success-outputs | Whether to include outputs of successful parallel tests ("true"/"false") | false | |
+| individual-providers-test-types-list-as-string | Which test types should be run for unit tests for providers (individually listed) | Providers[\amazon\] Providers\[google\] | * |
+| is-airflow-runner | Whether runner used is an airflow or infrastructure runner (true if airflow/false if infrastructure) | false | |
+| is-amd-runner | Whether runner used is an AMD one | true | |
+| is-arm-runner | Whether runner used is an ARM one | false | |
+| is-committer-build | Whether the build is triggered by a committer | false | |
+| is-k8s-runner | Whether the build runs on our k8s infrastructure | false | |
+| is-legacy-ui-api-labeled | Whether the PR is labeled as legacy UI/API | false | |
+| is-self-hosted-runner | Whether the runner is self-hosted | false | |
+| is-vm-runner | Whether the runner uses VM to run | true | |
+| kind-version | Which Kind version to use for tests | v0.24.0 | |
+| kubernetes-combos-list-as-string | All combinations of Python version and Kubernetes version to use for tests as space-separated string | 3.9-v1.25.2 3.10-v1.28.13 | * |
+| kubernetes-versions | All Kubernetes versions to use for tests as JSON array | \['v1.25.2'\] | |
+| kubernetes-versions-list-as-string | All Kubernetes versions to use for tests as space-separated string | v1.25.2 | * |
+| latest-versions-only | If set, the number of Python, Kubernetes, DB versions will be limited to the latest ones. | false | |
+| mypy-checks | List of folders to be considered for mypy checks | \["airflow_aux", "airflow_core"\] | |
+| mysql-exclude | Which versions of MySQL to exclude for tests as JSON array | [] | |
+| mysql-versions | Which versions of MySQL to use for tests as JSON array | \['8.0'\] | |
+| needs-api-codegen | Whether "api-codegen" are needed to run ("true"/"false") | true | |
+| needs-api-tests | Whether "api-tests" are needed to run ("true"/"false") | true | |
+| needs-helm-tests | Whether Helm tests are needed to run ("true"/"false") | true | |
+| needs-javascript-scans | Whether javascript CodeQL scans should be run ("true"/"false") | true | |
+| needs-mypy | Whether mypy check is supposed to run in this build | true | |
+| needs-python-scans | Whether Python CodeQL scans should be run ("true"/"false") | true | |
+| only-new-ui-files | Whether only new UI files are present in the PR ("true"/"false") | false | |
+| postgres-exclude | Which versions of Postgres to exclude for tests as JSON array | [] | |
+| postgres-versions | Which versions of Postgres to use for tests as JSON array | \['12'\] | |
+| prod-image-build | Whether PROD image build is needed | true | |
+| providers-compatibility-tests-matrix | Matrix of providers compatibility tests: (python_version, airflow_version, removed_providers) | \[{}\] | |
+| providers-test-types-list-as-string | Which test types should be run for unit tests for providers | Providers Providers\[-google\] | * |
+| pyproject-toml-changed | When pyproject.toml changed in the PR. | false | |
+| python-versions | List of python versions to use for that build | \['3.9'\] | |
+| python-versions-list-as-string | Which versions of MySQL to use for tests as space-separated string | 3.9 | * |
+| run-amazon-tests | Whether Amazon tests should be run ("true"/"false") | true | |
+| run-kubernetes-tests | Whether Kubernetes tests should be run ("true"/"false") | true | |
+| run-system-tests | Whether system tests should be run ("true"/"false") | true | |
+| run-task-sdk-tests | Whether Task SDK tests should be run ("true"/"false") | true | |
+| run-tests | Whether unit tests should be run ("true"/"false") | true | |
+| run-ui-tests | Whether UI tests should be run ("true"/"false") | true | |
+| run-www-tests | Whether Legacy WWW tests should be run ("true"/"false") | true | |
+| runs-on-as-json-default | List of labels assigned for runners for that build for default runs for that build (as string) | \["ubuntu-22.04"\] | |
+| runs-on-as-json-docs-build | List of labels assigned for runners for that build for ddcs build (as string) | \["ubuntu-22.04"\] | |
+| runs-on-as-json-self-hosted | List of labels assigned for runners for that build for self hosted runners | \["self-hosted", "Linux", "X64"\] | |
+| runs-on-as-json-self-hosted-asf | List of labels assigned for runners for that build for ASF self hosted runners | \["self-hosted", "Linux", "X64"\] | |
+| runs-on-as-json-public | List of labels assigned for runners for that build for public runners | \["ubuntu-22.04"\] | |
+| selected-providers-list-as-string | List of providers affected when they are selectively affected. | airbyte http | * |
+| skip-pre-commits | Which pre-commits should be skipped during the static-checks run | flynt,identity | |
+| skip-providers-tests | When provider tests should be skipped (on non-main branch or when no provider changes detected) | true | |
+| sqlite-exclude | Which versions of Sqlite to exclude for tests as JSON array | [] | |
+| test-groups | List of test groups that are valid for this run | \['core', 'providers'\] | |
+| testable-core-integrations | List of core integrations that are testable in the build as JSON array | \['celery', 'kerberos'\] | |
+| testable-providers-integrations | List of core integrations that are testable in the build as JSON array | \['mongo', 'kafka'\] | |
+| upgrade-to-newer-dependencies | Whether the image build should attempt to upgrade all dependencies (true/false or commit hash) | false | |
[1] Note for deciding if `full tests needed` mode is enabled and provider.yaml files.
@@ -250,23 +269,15 @@ That's why we do not base our `full tests needed` decision on changes in depende
from the `provider.yaml` files, but on `generated/provider_dependencies.json` and `pyproject.toml` files being
modified. This can be overridden by setting `full tests needed` label in the PR.
-## Committer vs. non-committer PRs
+## Committer vs. Non-committer PRs
There is a difference in how the CI jobs are run for committer and non-committer PRs from forks.
-Main reason is security - we do not want to run untrusted code on our infrastructure for self-hosted runners,
-but also we do not want to run unverified code during the `Build imaage` workflow, because that workflow has
-access to GITHUB_TOKEN that has access to write to the Github Registry of ours (which is used to cache
-images between runs). Also those images are build on self-hosted runners and we have to make sure that
-those runners are not used to (fore example) mine cryptocurrencies on behalf of the person who opened the
-pull request from their newly opened fork of airflow.
+The main reason is security; we do not want to run untrusted code on our infrastructure for self-hosted runners.
-This is why the `Build Images` workflow checks if the actor of the PR (GITHUB_ACTOR) is one of the committers,
-and if not, then workflows and scripts used to run image building are coming only from the ``target`` branch
-of the repository, where such scripts were reviewed and approved by the committers before being merged.
-
-This is controlled by `Selective checks <04_selective_checks.md>`__ that set appropriate output in
-the build-info job of the workflow (see`is-committer-build` to `true`) if the actor is in the committer's
-list and can be overridden by `non committer build` label in the PR.
+Currently there is no difference because we are not using `self-hosted` runners (until we implement `Action
+Runner Controller` but most of the jobs, committer builds will use "Self-hosted" runners by default,
+while non-committer builds will use "Public" runners. For committers, this can be overridden by setting the
+`use public runners` label in the PR.
## Changing behaviours of the CI runs by setting labels
@@ -316,12 +327,13 @@ This table summarizes the labels you can use on PRs to control the selective che
| debug ci resources | debug-ci-resources | If set, then debugging resources is enabled during parallel tests and you can see them. |
| default versions only | all-versions, *-versions-* | If set, the number of Python and Kubernetes, DB versions are limited to the default ones. |
| disable image cache | docker-cache | If set, the image cache is disables when building the image. |
+| force pip | force-pip | If set, the image build uses pip instead of uv. |
| full tests needed | full-tests-needed | If set, complete set of tests are run |
| include success outputs | include-success-outputs | If set, outputs of successful parallel tests are shown not only failed outputs. |
| latest versions only | *-versions-*, *-versions-* | If set, the number of Python, Kubernetes, DB versions will be limited to the latest ones. |
| non committer build | is-committer-build | If set, the scripts used for images are used from target branch for committers. |
| upgrade to newer dependencies | upgrade-to-newer-dependencies | If set to true (default false) then dependencies in the CI image build are upgraded. |
-| use public runners | runs-on-as-json-default | Force using public runners as default runners. |
+| use public runners | runs-on-as-json-public | Force using public runners as default runners. |
| use self-hosted runners | runs-on-as-json-default | Force using self-hosted runners as default runners. |
-----
diff --git a/dev/breeze/doc/ci/05_workflows.md b/dev/breeze/doc/ci/05_workflows.md
index b70a81ef64c..0c66505508f 100644
--- a/dev/breeze/doc/ci/05_workflows.md
+++ b/dev/breeze/doc/ci/05_workflows.md
@@ -24,11 +24,8 @@
- [CI run types](#ci-run-types)
- [Pull request run](#pull-request-run)
- [Canary run](#canary-run)
- - [Scheduled runs](#scheduled-runs)
- [Workflows](#workflows)
- - [Build Images Workflow](#build-images-workflow)
- - [Differences for main and release branches](#differences-for-main-and-release-branches)
- - [Committer vs. non-committer PRs](#committer-vs-non-committer-prs)
+ - [Differences for `main` and `v*-*-test` branches](#differences-for-main-and-v--test-branches)
- [Tests Workflow](#tests-workflow)
- [CodeQL scan](#codeql-scan)
- [Publishing documentation](#publishing-documentation)
@@ -86,214 +83,133 @@ run in the context of the "apache/airflow" repository and has WRITE
access to the GitHub Container Registry.
When the PR changes important files (for example `generated/provider_depdencies.json` or
-`pyproject.toml`), the PR is run in "upgrade to newer dependencies" mode - where instead
-of using constraints to build images, attempt is made to upgrade all dependencies to latest
-versions and build images with them. This way we check how Airflow behaves when the
+`pyproject.toml` or `hatch_build.py`), the PR is run in "upgrade to newer dependencies" mode -
+where instead of using constraints to build images, attempt is made to upgrade
+all dependencies to latest versions and build images with them. This way we check how Airflow behaves when the
dependencies are upgraded. This can also be forced by setting the `upgrade to newer dependencies`
label in the PR if you are a committer and want to force dependency upgrade.
## Canary run
-This workflow is triggered when a pull request is merged into the "main"
-branch or pushed to any of the "v2-\*-test" branches. The "Canary" run
+This workflow is triggered when a pull request is merged into the `main`
+branch or pushed to any of the `v*-*-test` branches. The `canary` run
aims to upgrade dependencies to their latest versions and promptly
pushes a preview of the CI/PROD image cache to the GitHub Registry. This
allows pull requests to quickly utilize the new cache, which is
particularly beneficial when the Dockerfile or installation scripts have
been modified. Even if some tests fail, this cache will already include
-the latest Dockerfile and scripts.Upon successful execution, the run
+the latest Dockerfile and scripts. Upon successful execution, the run
updates the constraint files in the "constraints-main" branch with the
latest constraints and pushes both the cache and the latest CI/PROD
images to the GitHub Registry.
-If the "Canary" build fails, it often indicates that a new version of
+If the `canary` build fails, it often indicates that a new version of
our dependencies is incompatible with the current tests or Airflow code.
Alternatively, it could mean that a breaking change has been merged into
-"main". Both scenarios require prompt attention from the maintainers.
+`main`. Both scenarios require prompt attention from the maintainers.
While a "broken main" due to our code should be fixed quickly, "broken
dependencies" may take longer to resolve. Until the tests pass, the
constraints will not be updated, meaning that regular PRs will continue
using the older version of dependencies that passed one of the previous
-"Canary" runs.
+`canary` runs.
-## Scheduled runs
-
-The "scheduled" workflow, which is designed to run regularly (typically
-overnight), is triggered when a scheduled run occurs. This workflow is
-largely identical to the "Canary" run, with one key difference: the
-image is always built from scratch, not from a cache. This approach
-ensures that we can verify whether any "system" dependencies in the
-Debian base image have changed, and confirm that the build process
-remains reproducible. Since the process for a scheduled run mirrors that
-of a "Canary" run, no separate diagram is necessary to illustrate it.
+The `canary` runs are executed 6 times a day on schedule, you can also
+trigger the `canary` run manually via `workflow-dispatch` mechanism.
# Workflows
-A general note about cancelling duplicated workflows: for the
-`Build Images`, `Tests` and `CodeQL` workflows we use the `concurrency`
-feature of GitHub actions to automatically cancel "old" workflow runs of
-each type -- meaning if you push a new commit to a branch or to a pull
-request and there is a workflow running, GitHub Actions will cancel the
-old workflow run automatically.
-
-## Build Images Workflow
-
-This workflow builds images for the CI Workflow for Pull Requests coming
-from forks.
-
-It's a special type of workflow: `pull_request_target` which means that
-it is triggered when a pull request is opened. This also means that the
-workflow has Write permission to push to the GitHub registry the images
-used by CI jobs which means that the images can be built only once and
-reused by all the CI jobs (including the matrix jobs). We've implemented
-it so that the `Tests` workflow waits until the images are built by the
-`Build Images` workflow before running.
-
-Those "Build Image" steps are skipped in case Pull Requests do not come
-from "forks" (i.e. those are internal PRs for Apache Airflow repository.
-This is because in case of PRs coming from Apache Airflow (only
-committers can create those) the "pull_request" workflows have enough
-permission to push images to GitHub Registry.
-
-This workflow is not triggered on normal pushes to our "main" branches,
-i.e. after a pull request is merged and whenever `scheduled` run is
-triggered. Again in this case the "CI" workflow has enough permissions
-to push the images. In this case we simply do not run this workflow.
-
-The workflow has the following jobs:
-
-| Job | Description |
-|-------------------|---------------------------------------------|
-| Build Info | Prints detailed information about the build |
-| Build CI images | Builds all configured CI images |
-| Build PROD images | Builds all configured PROD images |
-
-The images are stored in the [GitHub Container
-Registry](https://github.com/orgs/apache/packages?repo_name=airflow) and the names of those images follow the patterns
-described in [Images](02_images.md#naming-conventions)
+A general note about cancelling duplicated workflows: for `Tests` and `CodeQL` workflows,
+we use the `concurrency` feature of GitHub actions to automatically cancel "old" workflow runs of
+each type. This means that if you push a new commit to a branch or to a pull
+request while a workflow is already running, GitHub Actions will automatically cancel the
+old workflow run.
-Image building is configured in "fail-fast" mode. When any of the images
-fails to build, it cancels other builds and the source `Tests` workflow
-run that triggered it.
-
-## Differences for main and release branches
+## Differences for `main` and `v*-*-test` branches
The type of tests executed varies depending on the version or branch
-under test. For the "main" development branch, we run all tests to
+being tested. For the "main" development branch, we run all tests to
maintain the quality of Airflow. However, when releasing patch-level
-updates on older branches, we only run a subset of these tests. This is
-because older branches are exclusively used for releasing Airflow and
-its corresponding image, not for releasing providers or helm charts.
+updates on older branches, we only run a subset of tests. This is
+because older branches are used exclusively for releasing Airflow and
+its corresponding image, not for releasing providers or Helm charts,
+so all those tests are skipped there by default.
This behaviour is controlled by `default-branch` output of the
-build-info job. Whenever we create a branch for old version we update
+build-info job. Whenever we create a branch for an older version, we update
the `AIRFLOW_BRANCH` in `airflow_breeze/branch_defaults.py` to point to
-the new branch and there are a few places where selection of tests is
-based on whether this output is `main`. They are marked as - in the
-"Release branches" column of the table below.
-
-## Committer vs. non-committer PRs
-
-There is a difference in how the CI jobs are run for committer and non-committer PRs from forks.
-Main reason is security - we do not want to run untrusted code on our infrastructure for self-hosted runners,
-but also we do not want to run unverified code during the `Build imaage` workflow, because that workflow has
-access to GITHUB_TOKEN that has access to write to the Github Registry of ours (which is used to cache
-images between runs). Also those images are build on self-hosted runners and we have to make sure that
-those runners are not used to (fore example) mine cryptocurrencies on behalf of the person who opened the
-pull request from their newly opened fork of airflow.
-
-This is why the `Build Images` workflow checks if the actor of the PR (GITHUB_ACTOR) is one of the committers,
-and if not, then workflows and scripts used to run image building are coming only from the ``target`` branch
-of the repository, where such scripts were reviewed and approved by the committers before being merged.
-
... 31700 lines suppressed ...