You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2023/11/27 15:51:41 UTC
(airflow) branch build-common-io-from-sources updated (46f304df67 -> 6f274666a7)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch build-common-io-from-sources
in repository https://gitbox.apache.org/repos/asf/airflow.git
omit 46f304df67 Add feature to build "chicken-egg" packages from sources
omit c28ba46e13 Exclude common-io provider
omit 26990e2ed6 Update RELEASE_NOTES.rst
omit 4fad6e5862 Update version to 2.8.0
omit 9a4857bd87 Update default branches for 2-8
add 99b68e2db2 Add OpenLineage support to GcsOperators - Delete, Transform and TimeSpanTransform (#35838)
add c41088b922 Revert "Revert "Add v2-8 branches to codecov.yml and .asf.yaml (#35750)" (#35883)" (#35886)
add f3ddefccf6 Simplify helm lint pre-commit by using common pre-commit code (#35880)
add 5d74ffb320 Fix gcs listing - ensure blobs are loaded (#34919)
new 6f274666a7 Add feature to build "chicken-egg" packages from sources
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (46f304df67)
\
N -- N -- N refs/heads/build-common-io-from-sources (6f274666a7)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.asf.yaml | 3 +
README.md | 18 +--
RELEASE_NOTES.rst | 172 +--------------------
airflow/__init__.py | 2 +-
airflow/api_connexion/openapi/v1.yaml | 2 +-
airflow/providers/google/cloud/hooks/gcs.py | 2 +
airflow/providers/google/cloud/operators/gcs.py | 95 ++++++++++--
codecov.yml | 2 +
dev/breeze/src/airflow_breeze/branch_defaults.py | 4 +-
.../logging-monitoring/logging-tasks.rst | 2 +-
.../installation/supported-versions.rst | 2 +-
docs/docker-stack/README.md | 10 +-
.../customizing/own-requirements.sh | 2 +-
.../extending/add-airflow-configuration/Dockerfile | 2 +-
.../extending/add-apt-packages/Dockerfile | 2 +-
.../add-build-essential-extend/Dockerfile | 2 +-
.../extending/add-providers/Dockerfile | 2 +-
.../extending/add-pypi-packages/Dockerfile | 2 +-
.../extending/add-requirement-packages/Dockerfile | 2 +-
.../extending/custom-providers/Dockerfile | 2 +-
.../extending/embedding-dags/Dockerfile | 2 +-
.../extending/writable-directory/Dockerfile | 2 +-
docs/docker-stack/entrypoint.rst | 18 +--
generated/PYPI_README.md | 16 +-
images/breeze/output_ci-image_build.txt | 2 +-
images/breeze/output_prod-image_build.txt | 2 +-
...elease-management_install-provider-packages.txt | 2 +-
...release-management_verify-provider-packages.txt | 2 +-
images/breeze/output_shell.txt | 2 +-
images/breeze/output_start-airflow.txt | 2 +-
newsfragments/35460.significant.rst | 10 ++
scripts/ci/pre_commit/pre_commit_helm_lint.py | 35 +++--
.../ci/pre_commit/pre_commit_supported_versions.py | 2 +-
tests/providers/google/cloud/operators/test_gcs.py | 160 +++++++++++++++++++
34 files changed, 335 insertions(+), 252 deletions(-)
create mode 100644 newsfragments/35460.significant.rst
(airflow) 01/01: Add feature to build "chicken-egg" packages from sources
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch build-common-io-from-sources
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6f274666a7c374935f57efb2d86a96a5e5d102b3
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Mon Nov 27 15:48:43 2023 +0100
Add feature to build "chicken-egg" packages from sources
When we build on ci a pre-release version of provider, and we want
to include packages that have >= CURRENTLY_RELEASED_VERSION
we have to make sure that those packages are built from sources
during building of PROD image. Otherwise they will not be installable
on CURRENT_VERSION.dev0, CURRENT_VERSION.rc* etc.
Until we "Actually" release a provider we should have a way to
build such provider from sources.
This is the CI version of it, once we have it working we can also
apply it to the workflow that releases images to dockerhub.
---
.github/actions/build-prod-images/action.yml | 12 ++++++++++++
.github/workflows/build-images.yml | 2 ++
.github/workflows/ci.yml | 3 +++
dev/breeze/src/airflow_breeze/utils/packages.py | 4 ----
dev/breeze/src/airflow_breeze/utils/selective_checks.py | 10 ++++++++++
scripts/in_container/_in_container_utils.sh | 7 -------
6 files changed, 27 insertions(+), 11 deletions(-)
diff --git a/.github/actions/build-prod-images/action.yml b/.github/actions/build-prod-images/action.yml
index 5fdbb795c4..2e928e8bb3 100644
--- a/.github/actions/build-prod-images/action.yml
+++ b/.github/actions/build-prod-images/action.yml
@@ -22,6 +22,9 @@ inputs:
build-provider-packages:
description: 'Whether to build provider packages from sources'
required: true
+ chicken-egg-providers:
+ description: 'Whether to build chicken-egg packages'
+ required: true
runs:
using: "composite"
steps:
@@ -41,6 +44,15 @@ runs:
--package-list-file ./airflow/providers/installed_providers.txt
--package-format wheel --version-suffix-for-pypi dev0
if: ${{ inputs.build-provider-packages == 'true' }}
+ - name: "Prepare chicken-eggs provider packages"
+ # In case of provider packages which use latest dev0 version of providers, we should prepare them
+ # from the source code, not from the PyPI because they have apache-airflow>=X.Y.Z dependency
+ # And when we prepare them from sources they will have apache-airflow>=X.Y.Z.dev0
+ shell: bash
+ run: >
+ breeze release-management prepare-provider-packages
+ --package-format wheel --version-suffix-for-pypi dev0 ${{ inputs.chicken-egg-providers }}
+ if: ${{ inputs.build-provider-packages != 'true' && inputs.chicken-egg-providers != '' }}
- name: "Prepare airflow package"
shell: bash
run: >
diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml
index b29d49de17..82cc6e5987 100644
--- a/.github/workflows/build-images.yml
+++ b/.github/workflows/build-images.yml
@@ -74,6 +74,7 @@ jobs:
is-arm-runner: ${{ steps.selective-checks.outputs.is-arm-runner }}
is-vm-runner: ${{ steps.selective-checks.outputs.is-vm-runner }}
is-k8s-runner: ${{ steps.selective-checks.outputs.is-k8s-runner }}
+ chicken-egg-providers: ${{ steps.selective-checks.outputs.chicken-egg-providers }}
target-commit-sha: "${{steps.discover-pr-merge-commit.outputs.target-commit-sha ||
github.event.pull_request.head.sha ||
github.sha
@@ -293,6 +294,7 @@ jobs:
uses: ./.github/actions/build-prod-images
with:
build-provider-packages: ${{ needs.build-info.outputs.default-branch == 'main' }}
+ chicken-egg-providers: ${{ needs.build-info.outputs.chicken-egg-providers }}
env:
UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
DOCKER_CACHE: ${{ needs.build-info.outputs.cache-directive }}
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 53e4da6916..72ab1dbd8c 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -121,6 +121,7 @@ jobs:
is-arm-runner: ${{ steps.selective-checks.outputs.is-arm-runner }}
is-vm-runner: ${{ steps.selective-checks.outputs.is-vm-runner }}
is-k8s-runner: ${{ steps.selective-checks.outputs.is-k8s-runner }}
+ chicken-egg-providers: ${{ steps.selective-checks.outputs.chicken-egg-providers }}
has-migrations: ${{ steps.selective-checks.outputs.has-migrations }}
source-head-repo: ${{ steps.source-run-info.outputs.source-head-repo }}
pull-request-labels: ${{ steps.source-run-info.outputs.pr-labels }}
@@ -1608,6 +1609,7 @@ jobs:
if: needs.build-info.outputs.in-workflow-build == 'true'
with:
build-provider-packages: ${{ needs.build-info.outputs.default-branch == 'main' }}
+ chicken-egg-providers: ${{ needs.build-info.outputs.chicken-egg-providers }}
env:
UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
DOCKER_CACHE: ${{ needs.build-info.outputs.cache-directive }}
@@ -1647,6 +1649,7 @@ jobs:
uses: ./.github/actions/build-prod-images
with:
build-provider-packages: ${{ needs.build-info.outputs.default-branch == 'main' }}
+ chicken-egg-providers: ${{ needs.build-info.outputs.chicken-egg-providers }}
env:
UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
DOCKER_CACHE: ${{ needs.build-info.outputs.cache-directive }}
diff --git a/dev/breeze/src/airflow_breeze/utils/packages.py b/dev/breeze/src/airflow_breeze/utils/packages.py
index 3d59643dbc..5518fdebd7 100644
--- a/dev/breeze/src/airflow_breeze/utils/packages.py
+++ b/dev/breeze/src/airflow_breeze/utils/packages.py
@@ -56,10 +56,6 @@ LONG_PROVIDERS_PREFIX = "apache-airflow-providers-"
# TODO: use single source of truth for those
# for now we need to keep them in sync with the ones in setup.py
PREINSTALLED_PROVIDERS = [
- # Until we cut off the 2.8.0 branch and bump current airflow version to 2.9.0, we should
- # Keep common.io commented out in order ot be able to generate PyPI constraints because
- # The version from PyPI has requirement of apache-airflow>=2.8.0
- # "common.io",
"common.sql",
"ftp",
"http",
diff --git a/dev/breeze/src/airflow_breeze/utils/selective_checks.py b/dev/breeze/src/airflow_breeze/utils/selective_checks.py
index 0a751565ef..b9d194e67c 100644
--- a/dev/breeze/src/airflow_breeze/utils/selective_checks.py
+++ b/dev/breeze/src/airflow_breeze/utils/selective_checks.py
@@ -1019,3 +1019,13 @@ class SelectiveChecks:
@cached_property
def has_migrations(self) -> bool:
return any([file.startswith("airflow/migrations/") for file in self._files])
+
+ @cached_property
+ def chicken_egg_providers(self) -> str:
+ """Space separated list of providers with chicken-egg problem and should be built from sources."""
+ if self._default_branch != "main":
+ # TODO: This is temporary only - until we have packages that have >= UPCOMING_VERSION of Airflow
+ # Because packages released in PyPI have >= UPCOMING_VERSION rather than >= UPCOMING_VERSION.dev0
+ # Once the release is done, we should remove this and build package from PyPI
+ return "common.io"
+ return ""
diff --git a/scripts/in_container/_in_container_utils.sh b/scripts/in_container/_in_container_utils.sh
index a3e09e0f1e..f78e044420 100644
--- a/scripts/in_container/_in_container_utils.sh
+++ b/scripts/in_container/_in_container_utils.sh
@@ -235,13 +235,6 @@ function install_all_providers_from_pypi_with_eager_upgrade() {
local res
for provider_package in ${ALL_PROVIDERS_PACKAGES}
do
- # Remove common.io provider in main branch until we cut-off v2-8-test branch and change
- # version in main to 2.9.0 - otherwise we won't be able to generate PyPI constraints as
- # released common-io provider has apache-airflow>2.8.0 as dependency and we cannot install
- # the provider from PyPI
- if [[ ${provider_package} == "apache-airflow-providers-common-io" ]]; then
- continue
- fi
echo -n "Checking if ${provider_package} is available in PyPI: "
res=$(curl --head -s -o /dev/null -w "%{http_code}" "https://pypi.org/project/${provider_package}/")
if [[ ${res} == "200" ]]; then