[ANNOUNCE] Breeze caching improvements

[Jarek Potiuk <ja...@potiuk.com>]

Hello Everyone,

Tl;DR: If you are using Breeze. Your image rebuild experience might
become much better now. Also our CI builds **should** become faster
(and cheaper) in general (maybe needs another small fix).

What you need to do:

1) Rebase to latest main (this is very important - you should rebase
all the work you have)
2) Run `./breeze cleanup-image`
3) Run `./breeze build-image`

Let me know in #airlfow-breeze channel slack if this is working very
slow for you (it should not). The next time you run `./breeze
build-image` it should be as fast as it can get automatically
(including pulling image as needed). I still think there might be a
few edge cases, so I would like to help to solve them.

Context:

Recently we've enabled support for MacOS M1 people for development (I
can finally say it is REALLY possible to use M1 for developments and
Breeze) - but the last few days (or even weeks) I was looking at
various edge cases for image building and caching. The last few weeks
when you tried to rebuild Breeze image it could take a really long
time, but I believe I solved all the issues and merged a change that
will make it much faster on a daily basis  - for both Intel and ARM
based Linux/Mac (If you have an ARM linux, it **should** also work!)

As the old wisdom says: "There are 2 hard problems in computer
science: cache invalidation, naming things, and off-by-1 errors.". And
I can definitely attest to that. While the new Buildkit and caching is
great, making it all works with Multi-platform builds with a number of
edge cases was a real challenge. But I THINK I got it right finally.

J.

Re: [ANNOUNCE] Breeze caching improvements

[Daniel Standish <da...@astronomer.io.INVALID>]

awesome 👏

Re: [ANNOUNCE] Breeze caching improvements

[Jarek Potiuk <ja...@potiuk.com>]

OK. Our CI builds should be much snappier now. I've found the last
problem and instead of 10 minute few days ago most PRs will have the
images built from cache in ~1m10 s.

On Thu, Mar 31, 2022 at 2:30 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> Hello Everyone,
>
> Tl;DR: If you are using Breeze. Your image rebuild experience might
> become much better now. Also our CI builds **should** become faster
> (and cheaper) in general (maybe needs another small fix).
>
> What you need to do:
>
> 1) Rebase to latest main (this is very important - you should rebase
> all the work you have)
> 2) Run `./breeze cleanup-image`
> 3) Run `./breeze build-image`
>
> Let me know in #airlfow-breeze channel slack if this is working very
> slow for you (it should not). The next time you run `./breeze
> build-image` it should be as fast as it can get automatically
> (including pulling image as needed). I still think there might be a
> few edge cases, so I would like to help to solve them.
>
> Context:
>
> Recently we've enabled support for MacOS M1 people for development (I
> can finally say it is REALLY possible to use M1 for developments and
> Breeze) - but the last few days (or even weeks) I was looking at
> various edge cases for image building and caching. The last few weeks
> when you tried to rebuild Breeze image it could take a really long
> time, but I believe I solved all the issues and merged a change that
> will make it much faster on a daily basis  - for both Intel and ARM
> based Linux/Mac (If you have an ARM linux, it **should** also work!)
>
> As the old wisdom says: "There are 2 hard problems in computer
> science: cache invalidation, naming things, and off-by-1 errors.". And
> I can definitely attest to that. While the new Buildkit and caching is
> great, making it all works with Multi-platform builds with a number of
> edge cases was a real challenge. But I THINK I got it right finally.
>
> J.