[GitHub] [airflow] potiuk edited a comment on pull request #10368: CI Images are now pre-build and stored in registry

[GitBox <gi...@apache.org>]

potiuk edited a comment on pull request #10368:
URL: https://github.com/apache/airflow/pull/10368#issuecomment-689744391


   > I'm just trying to understand how this change works -- it's definitely a good feature.
   
   Sure, no problem it's all quite well described including the sequence diagrams and what happens in the [CI.rst](https://github.com/apache/airflow/blob/master/CI.rst#ci-sequence-diagrams). I think a look at the sequence diagrams can help to understand it.
   
   However indeed it's not straightforward how it works, and it's a long document with lots of details so here, the gist of it:
   
   For every Push/Pull Request/Schedule Workflow you have a corresponding "Build Image" run (which is of `workflow_run' type). The reason why they are separate, is that the "workflow_run" type of event has the "write" access token and can push the images to the registry. It uses the "master" version of workflow to run, so there is no risk that someone will inject their own code from PR into the build process, so "write access" is quite OK. In our case I make sure that we use "workflow" + "scripts"  from the master branch even if we use other sources (airflow sources that end up in the image) from the incoming PR commit. The "Build image" pushes the build to the registry and "CI Image" waits until the image is available in that registry.
   
   You can very easily find those builds (though there is no direct link in the original PR.  When you click on [Actions tab](https://github.com/apache/airflow/actions) - you will always see two builds next to each other "Build Images" one and the actual "PR build". They are always next to each other (because they are started at the same time). Also I made sure that you have an easy way to make sure which build matches which. Each "CI build" has a unique "run id" - and in both "Build Image" and "CI Build" workflows there is "Build Info" step that (when you click on it) show all the information necessary to match those (it's in job name to make it easier), The run id, commit hash of the commit, branch and repository from which the build was triggered is all there and you can be quite sure which build is which.
   
   Also, the "run_id" is used to identify the imge itself. If you look at the "Wait for image" it will show the "run_id" it waits for and the images in github registry are tagged with this Run id. For example you can see this in the logs of "waiting for image"
   
   ```
   Waiting for master-python3.6-build:246742927 image
   ```
   
   This also means that you can immediately locally reproduce any test failure - because the images are in the registry. It's enough to run this (if you see a failure in python 3.6 postgres core test for example):
   ```
   ./breeze --github-image-id 246742927 --python 3.6 --backend postgres 
   ```
   
   And after pulling the image you are dropped into the very same image that was used to run on CI and you can 100% reproduce the test that failed (barring heisentests and side effects of other tests). 
   
   I hope it's helpful. Let me know if you have any more questions on that.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org