You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/24 11:55:40 UTC
[GitHub] [airflow] feluelle commented on a change in pull request #13865: Easy switching between GitHub Container Registries
feluelle commented on a change in pull request #13865:
URL: https://github.com/apache/airflow/pull/13865#discussion_r563280400
##########
File path: IMAGES.rst
##########
@@ -226,7 +226,12 @@ Choosing image registry
=======================
By default images are pulled and pushed from and to DockerHub registry when you use Breeze's push-image
-or build commands.
+or build commands. But as described in `CI Documentaton <CI.rst>`_, you can choose different image
+registry by setting ``GITHUB_REGISTRY`` to ``docker.pkg.github.com`` for Github Package Registry or
+``ghcr.io`` for GitHub Container Registry.
+
+Default is the Github Package Registry one. The Pull Request forks have no access to the secret byt they
Review comment:
```suggestion
Default is the Github Package Registry one. The Pull Request forks have no access to the secret but they
```
##########
File path: IMAGES.rst
##########
@@ -347,6 +352,60 @@ GitHub Container Registry
docker login ghcr.io
+Interacting with container registries
+=====================================
+
+Since there are different naming conventions used for Airflow images and there are multiple images used,
+`Breeze <BREEZE.rst>`_ provides easy to use management interface for the images. The
+`CI system of ours <CI.rst>`_ is designed in the way that it should automatically refresh caches, rebuild
+the images periodically and update them whenever new version of base python is released.
+However, occasionally, you might need to rebuild images locally and push them directly to the registries
+to refresh them.
+
+This can be done with ``Breeze`` command line which has easy-to-use tool to manage those images. For
+example:
+
+
+Force building Python 3.6 CI image using local cache and pushing it container registry:
+
+.. code-block:: bash
+
+ ./breeze build-image --python 3.6 --force-build-images --build-cache-local
+ ./breeze push-image --python 3.6 --github-registry ghcr.io
+
+
+Building Python 3.7 PROD images (both build and final image) using cache pulled
+from ``docker.package.github.com`` and pushing it back:
+
+.. code-block:: bash
+
+ ./breeze build-image --production-image --python 3.7 --github-registry docker.package.github.com
+ ./breeze push-image --production-image --python 3.7 --github-registry docker.package.github.com
Review comment:
```suggestion
./breeze push-image --production-image --python 3.7 --github-registry docker.package.github.com
```
##########
File path: IMAGES.rst
##########
@@ -347,6 +352,60 @@ GitHub Container Registry
docker login ghcr.io
+Interacting with container registries
+=====================================
+
+Since there are different naming conventions used for Airflow images and there are multiple images used,
+`Breeze <BREEZE.rst>`_ provides easy to use management interface for the images. The
+`CI system of ours <CI.rst>`_ is designed in the way that it should automatically refresh caches, rebuild
+the images periodically and update them whenever new version of base python is released.
+However, occasionally, you might need to rebuild images locally and push them directly to the registries
+to refresh them.
+
+This can be done with ``Breeze`` command line which has easy-to-use tool to manage those images. For
+example:
+
+
+Force building Python 3.6 CI image using local cache and pushing it container registry:
+
+.. code-block:: bash
+
+ ./breeze build-image --python 3.6 --force-build-images --build-cache-local
+ ./breeze push-image --python 3.6 --github-registry ghcr.io
+
+
+Building Python 3.7 PROD images (both build and final image) using cache pulled
+from ``docker.package.github.com`` and pushing it back:
+
+.. code-block:: bash
+
+ ./breeze build-image --production-image --python 3.7 --github-registry docker.package.github.com
+ ./breeze push-image --production-image --python 3.7 --github-registry docker.package.github.com
+
+
+Building Python 3.8 CI image using cache pulled from DockerHub and pushing it back:
+
+.. code-block:: bash
+
+ ./breeze build-image --python 3.8
+ ./breeze push-image --python 3.8
Review comment:
```suggestion
./breeze push-image --python 3.8
```
##########
File path: CI.rst
##########
@@ -834,3 +888,51 @@ Scheduled build flow
.. image:: images/ci/scheduled_ci_flow.png
:align: center
:alt: Scheduled build flow
+
+
+Adding new Python versions to CI
+--------------------------------
+
+In 2.0 line we currently support Python 3.6, 3.7, 3.8.
+
+In order to add a new version the following operations should be done (example uses python 3.9)
+
+* copy the latest constraints in ``constraints-master`` branch from previous versions and name it
+ using the new Python version (``constraints-3.9.txt``). Commit and push
+
+* add the new python version to `breeze-complete <breeze-complete>`_ and
+ `_initialization.sh <scripts/ci/libraries/_initialization.sh>`_ - tests will fail if they are not
+ in sync.
+
+* build image locally for both prod and CI locally using Breeze:
+
+.. code-block:: bash
+
+ ./breeze build-image --python 3.9
+
+* push image as cache to DockerHub and both registries:
+
+.. code-block:: bash
+
+ ./breeze push-image --python 3.9
+ ./breeze push-image --python 3.9 --github-registry ghcr.io
+ ./breeze push-image --python 3.9 --github-registry docker.package.github.com
+
+* Find the 3 new images (main, ci, build) created in
+ `GitHub Container registry<https://github.com/orgs/apache/packages?tab=packages&ecosystem=container&q=airflow>`_
+ go to Package Settings and turn on ``Public Visibility`` and add ``airflow-committers``
+ group as ``Admin Role`` to all of them.
+
+* In `DockerHub <https://hub.docker.com/repository/docker/apache/airflow/builds/edit>`_ create three entries
+ for automatically built nightly-tag and releae images:
Review comment:
```suggestion
for automatically built nightly-tag and release images:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org