You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2021/06/22 19:25:25 UTC
[airflow] 47/47: Switch to GitHub Container Registry by default
(#16586)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 1c0df06a2f07ecc1652eba61ad6d05f856701ea2
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Tue Jun 22 15:30:31 2021 +0200
Switch to GitHub Container Registry by default (#16586)
Yesterday GitHub moved Github Container Registry to
General Availability status. We are prepared to switch and tested
it before, so this PR attempts to switch to it.
(cherry picked from commit f7aebeb8a1fb634ca7369a4455c7324f67a9deda)
---
.github/workflows/build-images.yml | 4 +--
.github/workflows/ci.yml | 4 +--
BREEZE.rst | 12 ++++-----
CI.rst | 47 +++++++--------------------------
IMAGES.rst | 2 +-
breeze-complete | 2 +-
scripts/ci/libraries/_build_images.sh | 23 ++--------------
scripts/ci/libraries/_initialization.sh | 2 +-
8 files changed, 22 insertions(+), 74 deletions(-)
diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml
index 8ccd7a6..9097241 100644
--- a/.github/workflows/build-images.yml
+++ b/.github/workflows/build-images.yml
@@ -44,12 +44,10 @@ env:
secrets.CONSTRAINTS_GITHUB_REPOSITORY || 'apache/airflow' }}
# This token is WRITE one - pull_request_target type of events always have the WRITE token
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- # This token should not be empty in pull_request_target type of event.
- CONTAINER_REGISTRY_TOKEN: ${{ secrets.PAT_CR }}
GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
GITHUB_REGISTRY_WAIT_FOR_IMAGE: "false"
INSTALL_PROVIDERS_FROM_SOURCES: "true"
- GITHUB_REGISTRY: ${{ secrets.OVERRIDE_GITHUB_REGISTRY }}
+ GITHUB_REGISTRY: "ghcr.io"
TARGET_COMMIT_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
concurrency:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 32cf003..12be9f9 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -37,8 +37,7 @@ env:
VERBOSE: "true"
DOCKER_CACHE: "pulled"
USE_GITHUB_REGISTRY: "true"
- # Might be either 'ghcr.io' or 'docker.pkg.github.com'
- GITHUB_REGISTRY: "docker.pkg.github.com"
+ GITHUB_REGISTRY: "ghcr.io"
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_USERNAME: ${{ github.actor }}
# You can override CONSTRAINTS_GITHUB_REPOSITORY by setting secret in your repo but by default the
@@ -50,7 +49,6 @@ env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# In builds from forks, this token is empty, and this is good because such builds do not even try
# to push images to the registry.
- CONTAINER_REGISTRY_TOKEN: ${{ secrets.PAT_CR }}
GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ github.event.pull_request.head.sha || github.sha }}"
GITHUB_REGISTRY_PUSH_IMAGE_TAG: "latest"
INSTALL_PROVIDERS_FROM_SOURCES: "true"
diff --git a/BREEZE.rst b/BREEZE.rst
index 4e13653..c05a3e7 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -1449,11 +1449,11 @@ This is the current syntax for `./breeze <./breeze>`_:
GitHub registry used. GitHub has legacy Packages registry and Public Beta Container
registry.
- Default: docker.pkg.github.com.
+ Default: ghcr.io.
If you use this flag, automatically --use-github-registry flag is enabled.
- docker.pkg.github.com ghcr.io
+ ghcr.io docker.pkg.github.com
-g, --github-repository GITHUB_REPOSITORY
GitHub repository used to pull, push images when cache is used.
@@ -1623,11 +1623,11 @@ This is the current syntax for `./breeze <./breeze>`_:
GitHub registry used. GitHub has legacy Packages registry and Public Beta Container
registry.
- Default: docker.pkg.github.com.
+ Default: ghcr.io.
If you use this flag, automatically --use-github-registry flag is enabled.
- docker.pkg.github.com ghcr.io
+ ghcr.io docker.pkg.github.com
-g, --github-repository GITHUB_REPOSITORY
GitHub repository used to pull, push images when cache is used.
@@ -2696,11 +2696,11 @@ This is the current syntax for `./breeze <./breeze>`_:
GitHub registry used. GitHub has legacy Packages registry and Public Beta Container
registry.
- Default: docker.pkg.github.com.
+ Default: ghcr.io.
If you use this flag, automatically --use-github-registry flag is enabled.
- docker.pkg.github.com ghcr.io
+ ghcr.io docker.pkg.github.com
-g, --github-repository GITHUB_REPOSITORY
GitHub repository used to pull, push images when cache is used.
diff --git a/CI.rst b/CI.rst
index 2634bf2..03c076c 100644
--- a/CI.rst
+++ b/CI.rst
@@ -90,42 +90,21 @@ We can use either of the two available GitHub Container registries as cache:
* The new `GitHub Container Registry <https://docs.github.com/en/packages/guides/about-github-container-registry>`_
which is in Public Beta, has many more features (including permission management, public access and
- image retention possibility). It has also the drawback (at least as of January 2020) that you need to
- have separate personal access token created as ``PAT_CR`` secret in your repository with write access
- to registry in order to make it works. You also have to manually manage permissions of the images,
- i.e. after creating images for the first time, you need to set their visibility to "Public" and
+ image retention possibility). Similarly as in case of GitHub Package Registry ``GITHUB_TOKEN`` is needed
+ to push to the repositories. You also have to manually manage permissions of the images,
+ i.e. after creating images for the first time, you need to set their visibility to ``Public`` and
add ``Admin`` permissions to group of people managing the images (in our case ``airflow-committers`` group).
This makes it not very suitable to use GitHub container registry if you want to run builds of Airflow
in your own forks (note - it does not affect pull requests from forks to Airflow).
Those two images have different naming schemas. See `Images documentation <IMAGES.rst>`_ for details.
-You can choose which registry should be used by the repository by setting ``OVERRIDE_GITHUB_REGISTRY`` secret
-to either ``docker.pkg.github.com`` for GitHub Package Registry or ``ghcr.io`` for GitHub Container Registry.
-Default is the GitHub Package Registry one. The Pull Request forks have no access to the secret but they
-auto-detect the registry used when they wait for the images.
-
You can interact with the GitHub Registry images (pull/push) via `Breeze <BREEZE.rst>`_ - you can
pass ``--github-registry`` flag with either ``docker.pkg.github.com`` for GitHub Package Registry or
``ghcr.io`` for GitHub Container Registry and pull/push operations will be performed using the chosen
registry, using appropriate naming convention. This allows building and pushing the images locally by
committers who have access to push/pull those images.
-
-GitHub Container Registry Token
--------------------------------
-
-Unlike GitHub Packages, GitHub Registry requires a personal access token added as ``PAT_CR`` secret in order
-to make it works. This token has to have "Registry Write" scope. Ideally you should not use a token
-of a person who has access to many repositories, because this token allows to write packages in
-ANY repository, where the person has write access (including private organisations). Ideally, you need to have
-a separate account with only access to that repository and generate Personal Access Token with Package
-Registry write permission for that Account. Discussion about setting up such account is opened at
-`ASF Jira <https://issues.apache.org/jira/projects/INFRA/issues/INFRA-20959>`_. More info about
-the token for GitHub Container Registry can be found
-`here <https://docs.github.com/en/packages/guides/migrating-to-github-container-registry-for-docker-images#authenticating-with-the-container-registry>`_
-
-
Locally replicating CI failures
-------------------------------
@@ -441,7 +420,7 @@ the model of permission management is not the same for Container Registry as it
| USE_GITHUB_REGISTRY | true | If set to "true", we interact with GitHub |
| | | Registry registry not the DockerHub one. |
+--------------------------------+---------------------------+----------------------------------------------+
-| GITHUB_REGISTRY | ``docker.pkg.github.com`` | Name of the GitHub registry to use. Can be |
+| GITHUB_REGISTRY | ``ghcr.io`` | Name of the GitHub registry to use. Can be |
| | | ``docker.pkg.github.com`` or ``ghcr.io`` |
+--------------------------------+---------------------------+----------------------------------------------+
| GITHUB_REPOSITORY | ``apache/airflow`` | Prefix of the image. It indicates which. |
@@ -456,13 +435,6 @@ the model of permission management is not the same for Container Registry as it
| | | and to WRITE token for direct pushes and |
| | | scheduled or workflow_run types of builds |
+--------------------------------+---------------------------+----------------------------------------------+
-| CONTAINER_REGISTRY_TOKEN | | Personal token to use to login to GitHub |
-| | | Container Registry. Should be retrieved |
-| | | from secret (in our case it is PAT_CR secret |
-| | | following example in GitHub documentation. |
-| | | Only set in push/scheduled/workflow_run |
-| | | type of build. |
-+--------------------------------+---------------------------+----------------------------------------------+
| GITHUB_REGISTRY_WAIT_FOR_IMAGE | ``false`` | Wait for the image to be available. This is |
| | | useful if commit SHA is used as pull tag |
+--------------------------------+---------------------------+----------------------------------------------+
@@ -481,8 +453,7 @@ the CI process. The default registry is set to "GitHub Packages", but we are tes
Container Registry. In case of GitHub Packages, authentication uses GITHUB_TOKEN mechanism. Authentication
is needed for both pushing the images (WRITE) and pulling them (READ) - which means that GitHub token
is used in "main" build (WRITE) and in fork builds (READ). For container registry, our images are
-Publicly Visible and we do not need any authentication to pull them so the CONTAINER_REGISTRY_TOKEN is
-only set in the "main" builds only ("Build Images" workflow).
+Publicly Visible and we do not need any authentication to pull them.
Dockerhub Variables
===================
@@ -519,7 +490,7 @@ The following components are part of the CI infrastructure
* **GA CRON trigger** - GitHub Actions CRON triggering our jobs
* **GA Workers** - virtual machines running our jobs at GitHub Actions (max 20 in parallel)
* **GitHub Private Image Registry**- image registry used as build cache for CI jobs.
- It is at https://docker.pkg.github.com/apache/airflow/airflow
+ It is at https://ghcr.io/apache/airflow/airflow
* **DockerHub Public Image Registry** - publicly available image registry at DockerHub.
It is at https://hub.docker.com/r/apache/airflow-ci
* **DockerHub Build Workers** - virtual machines running build jibs at DockerHub
@@ -822,9 +793,9 @@ For example knowing that the CI build was for commit ``cd27124534b46c9688a1d89e7
.. code-block:: bash
- docker pull docker.pkg.github.com/apache/airflow/main-python3.6-ci:cd27124534b46c9688a1d89e75fcd137ab5137e3
+ docker pull ghcr.io/apache/airflow-main-python3.6-ci:cd27124534b46c9688a1d89e75fcd137ab5137e3
- docker run -it docker.pkg.github.com/apache/airflow/main-python3.6-ci:cd27124534b46c9688a1d89e75fcd137ab5137e3
+ docker run -it ghcr.io/apache/airflow-main-python3.6-ci:cd27124534b46c9688a1d89e75fcd137ab5137e3
But you usually need to pass more variables and complex setup if you want to connect to a database or
@@ -834,7 +805,7 @@ cd27124534b46c9688a1d89e75fcd137ab5137e3, in python 3.8 environment you can run:
.. code-block:: bash
- ./breeze --github-image-id cd27124534b46c9688a1d89e75fcd137ab5137e3 --github-registry docker.pkg.github.com --python 3.8
+ ./breeze --github-image-id cd27124534b46c9688a1d89e75fcd137ab5137e3 --github-registry ghcr.io --python 3.8
You will be dropped into a shell with the exact version that was used during the CI run and you will
be able to run pytest tests manually, easily reproducing the environment that was used in CI. Note that in
diff --git a/IMAGES.rst b/IMAGES.rst
index d332b99..e38257f 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -350,7 +350,7 @@ Images with a commit SHA (built for pull requests and pushes)
.. code-block:: bash
- docker.pkg.github.com.io/apache-airflow/<BRANCH>-pythonX.Y-ci-v2:<COMMIT_SHA> - for CI images
+ docker.pkg.github.com/apache-airflow/<BRANCH>-pythonX.Y-ci-v2:<COMMIT_SHA> - for CI images
docker.pkg.github.com/apache-airflow/<BRANCH>-pythonX.Y-v2:<COMMIT_SHA> - for production images
docker.pkg.github.com/apache-airflow/<BRANCH>-pythonX.Y-build-v2:<COMMIT_SHA> - for production build stage
docker.pkg.github.com/apache-airflow/python-v2:X.Y-slim-buster-<COMMIT_SHA> - for base Python images
diff --git a/breeze-complete b/breeze-complete
index f4445da..aac5076 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -28,7 +28,7 @@ _breeze_allowed_backends="sqlite mysql postgres"
_breeze_allowed_integrations="cassandra kerberos mongo openldap pinot rabbitmq redis statsd trino all"
_breeze_allowed_generate_constraints_modes="source-providers pypi-providers no-providers"
# registrys is good here even if it is not correct english. We are adding s automatically to all variables
-_breeze_allowed_github_registrys="docker.pkg.github.com ghcr.io"
+_breeze_allowed_github_registrys="ghcr.io docker.pkg.github.com"
_breeze_allowed_kubernetes_modes="image"
_breeze_allowed_kubernetes_versions="v1.20.2 v1.19.7 v1.18.15"
_breeze_allowed_helm_versions="v3.2.4"
diff --git a/scripts/ci/libraries/_build_images.sh b/scripts/ci/libraries/_build_images.sh
index 3da10d6..099cf85 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -422,30 +422,11 @@ function build_images::get_docker_image_names() {
}
# If GitHub Registry is used, login to the registry using GITHUB_USERNAME and
-# either GITHUB_TOKEN or CONTAINER_REGISTRY_TOKEN depending on the registry.
-# In case Personal Access token is not set, skip logging in
+# GITHUB_TOKEN. In case Personal Access token is not set, skip logging in
# Also enable experimental features of docker (we need `docker manifest` command)
function build_images::configure_docker_registry() {
if [[ ${USE_GITHUB_REGISTRY} == "true" ]]; then
- local token=""
- if [[ "${GITHUB_REGISTRY}" == "ghcr.io" ]]; then
- # For now ghcr.io can only authenticate using Personal Access Token with package access scope.
- # There are plans to implement GITHUB_TOKEN authentication but this is not implemented yet
- token="${CONTAINER_REGISTRY_TOKEN=}"
- verbosity::print_info
- verbosity::print_info "Using CONTAINER_REGISTRY_TOKEN!"
- verbosity::print_info
- elif [[ "${GITHUB_REGISTRY}" == "docker.pkg.github.com" ]]; then
- token="${GITHUB_TOKEN}"
- verbosity::print_info
- verbosity::print_info "Using GITHUB_TOKEN!"
- verbosity::print_info
- else
- echo
- echo "${COLOR_RED}ERROR: Bad value of '${GITHUB_REGISTRY}'. Should be either 'ghcr.io' or 'docker.pkg.github.com'!${COLOR_RESET}"
- echo
- exit 1
- fi
+ local token="${GITHUB_TOKEN}"
if [[ -z "${token}" ]] ; then
verbosity::print_info
verbosity::print_info "Skip logging in to GitHub Registry. No Token available!"
diff --git a/scripts/ci/libraries/_initialization.sh b/scripts/ci/libraries/_initialization.sh
index bf5d1eb..e8b3c91 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -545,7 +545,7 @@ function initialization::initialize_github_variables() {
# Defaults for interacting with GitHub
export USE_GITHUB_REGISTRY=${USE_GITHUB_REGISTRY:="false"}
export GITHUB_REGISTRY_IMAGE_SUFFIX=${GITHUB_REGISTRY_IMAGE_SUFFIX:="-v2"}
- export GITHUB_REGISTRY=${GITHUB_REGISTRY:="docker.pkg.github.com"}
+ export GITHUB_REGISTRY=${GITHUB_REGISTRY:="ghcr.io"}
export GITHUB_REGISTRY_WAIT_FOR_IMAGE=${GITHUB_REGISTRY_WAIT_FOR_IMAGE:="false"}
export GITHUB_REGISTRY_PULL_IMAGE_TAG=${GITHUB_REGISTRY_PULL_IMAGE_TAG:="latest"}
export GITHUB_REGISTRY_PUSH_IMAGE_TAG=${GITHUB_REGISTRY_PUSH_IMAGE_TAG:="latest"}