[GitHub] [airflow] potiuk edited a comment on pull request #9162: Improve production image iteration speed

[GitBox <gi...@apache.org>]

potiuk edited a comment on pull request #9162:
URL: https://github.com/apache/airflow/pull/9162#issuecomment-640104246


   Hey @ashb @dimberman @schnie @jhtimmins I think you  might want to merge that one :).
   
   It is on the biggish side and I can split it into few independent PRs if you think that it makes sense, but I think this one is crucial to get fast iteration on the production image while we are integrating the helm chart with it. Yesterday when @ashb worked on the chart we had slack conversation and I could help fairly quickly in adding some changes to the entrypoint, i saw later @ashb mentioned lack of clear-logs as another blocker. So I realised we need to have a way so that not only me but all of us can actually build and release new images quickly. I am happy to help (and I have a bunch of changes in the Github issues but since we work sometimes in different timezones, I think it is important to get iterations quicker.
   
   This change implements a number of related improvements to Breeze to make it super easy to iterat with production images  - including production images from released version of Airflow - and allows to quickly release patched versions. I already released -1 yesterday for @ash and I am just pushing -2 with clear-logs.
   
   You can take a look at the IMAGES.rst changes where I described everything but for your use it should be as easy as:
   
   `./breeze --python 3.7 build-image --install-airflow-version 1.10.10 --production-image --use-local-cache`
   
   That's it. This will build production image out of PyPi 1.10.10 with the local Dockerfile and scripts - so you can easily add new scripts, use them in Dockerfile and build production image locally. Then you can re-tag and push them manually. 
   
   What's super important - the PROD image is now optimized for BOTH - speed of rebuilds AND size.  If you use the `--use-local-cache` switch it will rebuild very quickly next time you build it - even if you modify setup.py and add new dependencies. For me it's < 20-30 seconds when I tried. I think it removes the biggest obstacle when working on the prod image - i.e. how quickly you can iterate.
   
   There are more changes in there for example once you build the prod image you can enter:
   
   `./breeze --production-image --python 3.7 shell bash` -> and you will be dropped in bash in the production image
   
   `./breeze --production-image --python 3.7 shell python` -> and you will be dropped in pyhton in the production image
   
   And few more things that I described in the description of the commit. 
   
   I also added "1.10.10-1, -2"  "release notes as well in the IMAGES.rst so that we can keep track of the changes while we work on them.
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org