You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/23 19:24:33 UTC
[GitHub] [airflow] potiuk opened a new pull request #13865: Easy switching between GitHub Container Registries
potiuk opened a new pull request #13865:
URL: https://github.com/apache/airflow/pull/13865
This change enables easy switching between GitHub Package Registry
and GitHub Container Registry by simply adding GITHUB_REGISTRY
secret to be either `docker.package.github.com` or `ghcr.io`.
This makes it easy to switch only by the Apache Airflow repository
run builds, as it requires preparation of images (to make them
public and to add permissions to manage them) after they got
created for the first time. GitHub Package Registry works
out-of-the-box but it is less stable and considered a legacy,
also it does not allow image retention.
Documentation has been updated to reflect the reasoning of choosing
this solution as well as describing maintenance processes around
images (including adding new Python version)
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #13865: Easy switching between GitHub Container Registries
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #13865:
URL: https://github.com/apache/airflow/pull/13865#issuecomment-774656623
Closing in favour of #14120
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #13865: Easy switching between GitHub Container Registries
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #13865:
URL: https://github.com/apache/airflow/pull/13865
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #13865: Easy switching between GitHub Container Registries
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13865:
URL: https://github.com/apache/airflow/pull/13865#issuecomment-766428417
[The Workflow run](https://github.com/apache/airflow/actions/runs/507872000) is cancelling this PR. Building images for the PR has failed. Follow the the workflow link to check the reason.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on a change in pull request #13865: Easy switching between GitHub Container Registries
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #13865:
URL: https://github.com/apache/airflow/pull/13865#discussion_r563280400
##########
File path: IMAGES.rst
##########
@@ -226,7 +226,12 @@ Choosing image registry
=======================
By default images are pulled and pushed from and to DockerHub registry when you use Breeze's push-image
-or build commands.
+or build commands. But as described in `CI Documentaton <CI.rst>`_, you can choose different image
+registry by setting ``GITHUB_REGISTRY`` to ``docker.pkg.github.com`` for Github Package Registry or
+``ghcr.io`` for GitHub Container Registry.
+
+Default is the Github Package Registry one. The Pull Request forks have no access to the secret byt they
Review comment:
```suggestion
Default is the Github Package Registry one. The Pull Request forks have no access to the secret but they
```
##########
File path: IMAGES.rst
##########
@@ -347,6 +352,60 @@ GitHub Container Registry
docker login ghcr.io
+Interacting with container registries
+=====================================
+
+Since there are different naming conventions used for Airflow images and there are multiple images used,
+`Breeze <BREEZE.rst>`_ provides easy to use management interface for the images. The
+`CI system of ours <CI.rst>`_ is designed in the way that it should automatically refresh caches, rebuild
+the images periodically and update them whenever new version of base python is released.
+However, occasionally, you might need to rebuild images locally and push them directly to the registries
+to refresh them.
+
+This can be done with ``Breeze`` command line which has easy-to-use tool to manage those images. For
+example:
+
+
+Force building Python 3.6 CI image using local cache and pushing it container registry:
+
+.. code-block:: bash
+
+ ./breeze build-image --python 3.6 --force-build-images --build-cache-local
+ ./breeze push-image --python 3.6 --github-registry ghcr.io
+
+
+Building Python 3.7 PROD images (both build and final image) using cache pulled
+from ``docker.package.github.com`` and pushing it back:
+
+.. code-block:: bash
+
+ ./breeze build-image --production-image --python 3.7 --github-registry docker.package.github.com
+ ./breeze push-image --production-image --python 3.7 --github-registry docker.package.github.com
Review comment:
```suggestion
./breeze push-image --production-image --python 3.7 --github-registry docker.package.github.com
```
##########
File path: IMAGES.rst
##########
@@ -347,6 +352,60 @@ GitHub Container Registry
docker login ghcr.io
+Interacting with container registries
+=====================================
+
+Since there are different naming conventions used for Airflow images and there are multiple images used,
+`Breeze <BREEZE.rst>`_ provides easy to use management interface for the images. The
+`CI system of ours <CI.rst>`_ is designed in the way that it should automatically refresh caches, rebuild
+the images periodically and update them whenever new version of base python is released.
+However, occasionally, you might need to rebuild images locally and push them directly to the registries
+to refresh them.
+
+This can be done with ``Breeze`` command line which has easy-to-use tool to manage those images. For
+example:
+
+
+Force building Python 3.6 CI image using local cache and pushing it container registry:
+
+.. code-block:: bash
+
+ ./breeze build-image --python 3.6 --force-build-images --build-cache-local
+ ./breeze push-image --python 3.6 --github-registry ghcr.io
+
+
+Building Python 3.7 PROD images (both build and final image) using cache pulled
+from ``docker.package.github.com`` and pushing it back:
+
+.. code-block:: bash
+
+ ./breeze build-image --production-image --python 3.7 --github-registry docker.package.github.com
+ ./breeze push-image --production-image --python 3.7 --github-registry docker.package.github.com
+
+
+Building Python 3.8 CI image using cache pulled from DockerHub and pushing it back:
+
+.. code-block:: bash
+
+ ./breeze build-image --python 3.8
+ ./breeze push-image --python 3.8
Review comment:
```suggestion
./breeze push-image --python 3.8
```
##########
File path: CI.rst
##########
@@ -834,3 +888,51 @@ Scheduled build flow
.. image:: images/ci/scheduled_ci_flow.png
:align: center
:alt: Scheduled build flow
+
+
+Adding new Python versions to CI
+--------------------------------
+
+In 2.0 line we currently support Python 3.6, 3.7, 3.8.
+
+In order to add a new version the following operations should be done (example uses python 3.9)
+
+* copy the latest constraints in ``constraints-master`` branch from previous versions and name it
+ using the new Python version (``constraints-3.9.txt``). Commit and push
+
+* add the new python version to `breeze-complete <breeze-complete>`_ and
+ `_initialization.sh <scripts/ci/libraries/_initialization.sh>`_ - tests will fail if they are not
+ in sync.
+
+* build image locally for both prod and CI locally using Breeze:
+
+.. code-block:: bash
+
+ ./breeze build-image --python 3.9
+
+* push image as cache to DockerHub and both registries:
+
+.. code-block:: bash
+
+ ./breeze push-image --python 3.9
+ ./breeze push-image --python 3.9 --github-registry ghcr.io
+ ./breeze push-image --python 3.9 --github-registry docker.package.github.com
+
+* Find the 3 new images (main, ci, build) created in
+ `GitHub Container registry<https://github.com/orgs/apache/packages?tab=packages&ecosystem=container&q=airflow>`_
+ go to Package Settings and turn on ``Public Visibility`` and add ``airflow-committers``
+ group as ``Admin Role`` to all of them.
+
+* In `DockerHub <https://hub.docker.com/repository/docker/apache/airflow/builds/edit>`_ create three entries
+ for automatically built nightly-tag and releae images:
Review comment:
```suggestion
for automatically built nightly-tag and release images:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #13865: Easy switching between GitHub Container Registries
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13865:
URL: https://github.com/apache/airflow/pull/13865#issuecomment-766428417
[The Workflow run](https://github.com/apache/airflow/actions/runs/507872000) is cancelling this PR. Building images for the PR has failed. Follow the the workflow link to check the reason.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #13865: Easy switching between GitHub Container Registries
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #13865:
URL: https://github.com/apache/airflow/pull/13865#issuecomment-766167201
Allows to switch very easily between GCR/Packages + all the documentation updated explaining reasoning why we do both and why and how they are used including how to add new images, how to manage them and how to conifgure Personal Access token.
@ashb - let me know if the `reasoning` i put there is the kind you wanted when we dicussed it last time - if you miss anything, please let me know, I will be happy to add it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org