You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/18 10:05:11 UTC
[GitHub] [airflow] potiuk commented on a change in pull request #10368: CI Images are now pre-build and stored in registry
potiuk commented on a change in pull request #10368:
URL: https://github.com/apache/airflow/pull/10368#discussion_r472064881
##########
File path: CI.rst
##########
@@ -188,32 +568,104 @@ The issues that gets updated are different for different branches:
* master: `Quarantine tests master <https://github.com/apache/airflow/issues/10118>`_
* v1-10-stable: `Quarantine tests v1-10-stable <https://github.com/apache/airflow/issues/10127>`_
-* v1-10-test: `Quarantine tests v1-10-test <hhttps://github.com/apache/airflow/issues/10128>`_
-
-+---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
-| Job | Description | Pull Request Run | Direct Push/Merge Run | Scheduled Run |
-+===========================+================================================================================================================+====================================+=================================+======================================================================+
-| Cancel previous workflow | Cancels the previously running workflow run if there is one running | Yes | Yes | Yes * |
-+---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
-| Trigger tests | Checks if tests should be triggered | Yes | Yes | Yes * |
-+---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
-| Quarantined tests | Those are tests that are flaky and we need to fix them | Yes (if tests-triggered) | Yes (Updates quarantine issue) | Yes * (updates quarantine issue) |
-+---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
-
-Cancel other workflow runs workflow
------------------------------------
-
-This workflow is run only on schedule (every 5 minutes) it's only purpose is to cancel other running
-``CI Build`` workflows if important jobs failed in those runs. This is to save runners for other runs
-in case we know that the build will not succeed anyway without some basic fixes to static checks or
-documentation - effectively implementing missing "fail-fast" (on a job level) in Github Actions
-similar to fail-fast in matrix strategy.
-
-The jobs that are considered as "fail-fast" are:
-
-* Static checks
-* Docs
-* Prepare Backport packages
-* Helm tests
-* Build Prod Image
-* TTest OpenAPI client gen
+* v1-10-test: `Quarantine tests v1-10-test <https://github.com/apache/airflow/issues/10128>`_
+
+Force sync master from apache/airflow
+-------------------------------------
+
+This is manually triggered workflow (via GitHub UI manual run) that should only be run in GitHub forks.
+When triggered, it will force-push the "apache/airflow" master to the fork's master. It's the easiest
+way to sync your fork master to the Apache Airflow's one.
+
+Naming conventions for stored images
+====================================
+
+The images produced during the CI builds are stored in the
+`GitHub Registry <https://github.com/apache/airflow/packages>`_
+
+The images are stored with both "latest" tag (for last master push image that passes all the tests as well
+with the tags indicating the origin of the image.
+
+The image names follow the patterns:
+
++--------------+----------------------------+--------------------------------+--------------------------------------------------------------------------------------------+
+| Image | Name pattern | Tag for format | Comment |
++==============+============================+================================+============================================================================================+
+| Python image | python | <X.Y>-slim-buster-<RUN_ID> | Base python image used by both production and CI image. |
+| | | <X.Y>-slim-buster-<COMMIT_SHA> | Python maintainer release new versions of those image with security fixes every few weeks. |
++--------------+----------------------------+--------------------------------+--------------------------------------------------------------------------------------------+
+| CI image | <BRANCH>-python<X.Y>-ci | <RUN_ID> | CI image - this is the image used for most of the tests. |
Review comment:
yeah. I wanted to introduce some retention after I got it up and running. Just did not want to make the change any bigger than it is. This is not as bad as it seems because most of these images will be just adding few last layers (mostly changed android sources) because we are always building incrementally using the previous images with --cache-from.
Until python releases a new version of base python image, 9X% of each image is the same - we simply add new layers on top. You can actually see it in my fork where most of the images we push are "Layer already exists".
But indeed the scheduled build always produces a new image built from scratch and every time we change Dockerfile at the beginning or python releases a new base image we have new layers updated. So yeah. I plan to delete the old tags (> week I think should be enough).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org