[ANNOUNCE] Speed-up of image releases

[Jarek Potiuk <ja...@potiuk.com>]

Hello here,

*TL;DR; Pushing release images to DockerHub is going to be more than 5x
faster (<15 minutes rather than way above 1hr).*

I am not sure if you are aware but for the 3.0.0 release, together with
Kaxil, we had to bend ourselves backwards a bit and do a few last minute,
high-stress bug-fixes to be able to release airflow images for testing -
bit RC and final images.

Also hopefully next releases will be way smoother than the first 3.0.0
releases - because with https://github.com/apache/airflow/pull/50176 we
could use  recent `ubuntu-22.04-arm` runners that GitHub made available. I
had to add capability of building the images separately and merging them to
make it work - but it turned out to be relatively easy - just incremental
addition to the existing breeze tooling.

With a few earlier PRs where I added a number of robustness and tests and
tested the scenarios where we had Alpha/Beta/RC packages and dependent
airflow Alpha/Beta/RC in various combinations and inter-dependencies of
those, and allowing various stages of pre-release for them and I think
finally we should have pretty robust way of being able to release Airflow
RC candidates depending on any release candidate of providers in pretty
much all reasonable combinations. There are some very early checks and
validations that should warn the Release Manager very early in case there
are any problems with the RC candidate.

I also tested the workflow when - if we find any fixable "dockerfile" or
"script" issue during the release we should be able to release images using
a completely separate branch with fixes.

This should speed up release image preparation to < 15 mins from more than
an hour and that also means that when the release manager announces
Airflow, images should be long ready then.

I hope that will make our future release process and testing way smoother.

J.

Re: [ANNOUNCE] Speed-up of image releases

[Jarek Potiuk <ja...@potiuk.com>]

And we have ARM tests now as well as a follow-up :).

And ... Some more speed ups are on the way .

The rumour is... (Pavan is leading that :) ) we are **just** going to have
the first release of our docs through - hopefully - much faster S3
publishing mechanism.
The current way of Providers is the first one to try it.

If all goes well - is also going to speed up doc publishing from > 30
minutes to < 5 minutes. And finally our `sites` repo will be "normal" size.

But :fingers crossed: ... We'll announce it when it works.

J.

On Tue, May 6, 2025 at 10:49 AM Kaxil Naik <ka...@gmail.com> wrote:

> Nice -- in time for 3.1 and was happy to see this coming out of long
> weekend.
>
> On Mon, 5 May 2025 at 17:49, Amogh Desai <am...@gmail.com> wrote:
>
> > This is awesome, thank you so much Jarek!
> >
> > This will significantly speed up things :)
> > Thanks & Regards,
> > Amogh Desai
> >
> >
> > On Mon, May 5, 2025 at 7:44 AM Pavankumar Gopidesu <
> > gopidesupavan@gmail.com>
> > wrote:
> >
> > > Woohoo that's huge 👏, thank you Jarek.
> > >
> > > Pavan.
> > >
> > > On Sun, May 4, 2025, 21:52 Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > Hello here,
> > > >
> > > > *TL;DR; Pushing release images to DockerHub is going to be more than
> 5x
> > > > faster (<15 minutes rather than way above 1hr).*
> > > >
> > > > I am not sure if you are aware but for the 3.0.0 release, together
> with
> > > > Kaxil, we had to bend ourselves backwards a bit and do a few last
> > minute,
> > > > high-stress bug-fixes to be able to release airflow images for
> testing
> > -
> > > > bit RC and final images.
> > > >
> > > > Also hopefully next releases will be way smoother than the first
> 3.0.0
> > > > releases - because with https://github.com/apache/airflow/pull/50176
> > we
> > > > could use  recent `ubuntu-22.04-arm` runners that GitHub made
> > available.
> > > I
> > > > had to add capability of building the images separately and merging
> > them
> > > to
> > > > make it work - but it turned out to be relatively easy - just
> > incremental
> > > > addition to the existing breeze tooling.
> > > >
> > > > With a few earlier PRs where I added a number of robustness and tests
> > and
> > > > tested the scenarios where we had Alpha/Beta/RC packages and
> dependent
> > > > airflow Alpha/Beta/RC in various combinations and inter-dependencies
> of
> > > > those, and allowing various stages of pre-release for them and I
> think
> > > > finally we should have pretty robust way of being able to release
> > Airflow
> > > > RC candidates depending on any release candidate of providers in
> pretty
> > > > much all reasonable combinations. There are some very early checks
> and
> > > > validations that should warn the Release Manager very early in case
> > there
> > > > are any problems with the RC candidate.
> > > >
> > > > I also tested the workflow when - if we find any fixable "dockerfile"
> > or
> > > > "script" issue during the release we should be able to release images
> > > using
> > > > a completely separate branch with fixes.
> > > >
> > > > This should speed up release image preparation to < 15 mins from more
> > > than
> > > > an hour and that also means that when the release manager announces
> > > > Airflow, images should be long ready then.
> > > >
> > > > I hope that will make our future release process and testing way
> > > smoother.
> > > >
> > > > J.
> > > >
> > >
> >
>

Re: [ANNOUNCE] Speed-up of image releases

[Kaxil Naik <ka...@gmail.com>]

Nice -- in time for 3.1 and was happy to see this coming out of long
weekend.

On Mon, 5 May 2025 at 17:49, Amogh Desai <am...@gmail.com> wrote:

> This is awesome, thank you so much Jarek!
>
> This will significantly speed up things :)
> Thanks & Regards,
> Amogh Desai
>
>
> On Mon, May 5, 2025 at 7:44 AM Pavankumar Gopidesu <
> gopidesupavan@gmail.com>
> wrote:
>
> > Woohoo that's huge 👏, thank you Jarek.
> >
> > Pavan.
> >
> > On Sun, May 4, 2025, 21:52 Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > Hello here,
> > >
> > > *TL;DR; Pushing release images to DockerHub is going to be more than 5x
> > > faster (<15 minutes rather than way above 1hr).*
> > >
> > > I am not sure if you are aware but for the 3.0.0 release, together with
> > > Kaxil, we had to bend ourselves backwards a bit and do a few last
> minute,
> > > high-stress bug-fixes to be able to release airflow images for testing
> -
> > > bit RC and final images.
> > >
> > > Also hopefully next releases will be way smoother than the first 3.0.0
> > > releases - because with https://github.com/apache/airflow/pull/50176
> we
> > > could use  recent `ubuntu-22.04-arm` runners that GitHub made
> available.
> > I
> > > had to add capability of building the images separately and merging
> them
> > to
> > > make it work - but it turned out to be relatively easy - just
> incremental
> > > addition to the existing breeze tooling.
> > >
> > > With a few earlier PRs where I added a number of robustness and tests
> and
> > > tested the scenarios where we had Alpha/Beta/RC packages and dependent
> > > airflow Alpha/Beta/RC in various combinations and inter-dependencies of
> > > those, and allowing various stages of pre-release for them and I think
> > > finally we should have pretty robust way of being able to release
> Airflow
> > > RC candidates depending on any release candidate of providers in pretty
> > > much all reasonable combinations. There are some very early checks and
> > > validations that should warn the Release Manager very early in case
> there
> > > are any problems with the RC candidate.
> > >
> > > I also tested the workflow when - if we find any fixable "dockerfile"
> or
> > > "script" issue during the release we should be able to release images
> > using
> > > a completely separate branch with fixes.
> > >
> > > This should speed up release image preparation to < 15 mins from more
> > than
> > > an hour and that also means that when the release manager announces
> > > Airflow, images should be long ready then.
> > >
> > > I hope that will make our future release process and testing way
> > smoother.
> > >
> > > J.
> > >
> >
>

Re: [ANNOUNCE] Speed-up of image releases

[Amogh Desai <am...@gmail.com>]

This is awesome, thank you so much Jarek!

This will significantly speed up things :)
Thanks & Regards,
Amogh Desai


On Mon, May 5, 2025 at 7:44 AM Pavankumar Gopidesu <go...@gmail.com>
wrote:

> Woohoo that's huge 👏, thank you Jarek.
>
> Pavan.
>
> On Sun, May 4, 2025, 21:52 Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Hello here,
> >
> > *TL;DR; Pushing release images to DockerHub is going to be more than 5x
> > faster (<15 minutes rather than way above 1hr).*
> >
> > I am not sure if you are aware but for the 3.0.0 release, together with
> > Kaxil, we had to bend ourselves backwards a bit and do a few last minute,
> > high-stress bug-fixes to be able to release airflow images for testing -
> > bit RC and final images.
> >
> > Also hopefully next releases will be way smoother than the first 3.0.0
> > releases - because with https://github.com/apache/airflow/pull/50176 we
> > could use  recent `ubuntu-22.04-arm` runners that GitHub made available.
> I
> > had to add capability of building the images separately and merging them
> to
> > make it work - but it turned out to be relatively easy - just incremental
> > addition to the existing breeze tooling.
> >
> > With a few earlier PRs where I added a number of robustness and tests and
> > tested the scenarios where we had Alpha/Beta/RC packages and dependent
> > airflow Alpha/Beta/RC in various combinations and inter-dependencies of
> > those, and allowing various stages of pre-release for them and I think
> > finally we should have pretty robust way of being able to release Airflow
> > RC candidates depending on any release candidate of providers in pretty
> > much all reasonable combinations. There are some very early checks and
> > validations that should warn the Release Manager very early in case there
> > are any problems with the RC candidate.
> >
> > I also tested the workflow when - if we find any fixable "dockerfile" or
> > "script" issue during the release we should be able to release images
> using
> > a completely separate branch with fixes.
> >
> > This should speed up release image preparation to < 15 mins from more
> than
> > an hour and that also means that when the release manager announces
> > Airflow, images should be long ready then.
> >
> > I hope that will make our future release process and testing way
> smoother.
> >
> > J.
> >
>

Re: [ANNOUNCE] Speed-up of image releases

[Pavankumar Gopidesu <go...@gmail.com>]

Woohoo that's huge 👏, thank you Jarek.

Pavan.

On Sun, May 4, 2025, 21:52 Jarek Potiuk <ja...@potiuk.com> wrote:

> Hello here,
>
> *TL;DR; Pushing release images to DockerHub is going to be more than 5x
> faster (<15 minutes rather than way above 1hr).*
>
> I am not sure if you are aware but for the 3.0.0 release, together with
> Kaxil, we had to bend ourselves backwards a bit and do a few last minute,
> high-stress bug-fixes to be able to release airflow images for testing -
> bit RC and final images.
>
> Also hopefully next releases will be way smoother than the first 3.0.0
> releases - because with https://github.com/apache/airflow/pull/50176 we
> could use  recent `ubuntu-22.04-arm` runners that GitHub made available. I
> had to add capability of building the images separately and merging them to
> make it work - but it turned out to be relatively easy - just incremental
> addition to the existing breeze tooling.
>
> With a few earlier PRs where I added a number of robustness and tests and
> tested the scenarios where we had Alpha/Beta/RC packages and dependent
> airflow Alpha/Beta/RC in various combinations and inter-dependencies of
> those, and allowing various stages of pre-release for them and I think
> finally we should have pretty robust way of being able to release Airflow
> RC candidates depending on any release candidate of providers in pretty
> much all reasonable combinations. There are some very early checks and
> validations that should warn the Release Manager very early in case there
> are any problems with the RC candidate.
>
> I also tested the workflow when - if we find any fixable "dockerfile" or
> "script" issue during the release we should be able to release images using
> a completely separate branch with fixes.
>
> This should speed up release image preparation to < 15 mins from more than
> an hour and that also means that when the release manager announces
> Airflow, images should be long ready then.
>
> I hope that will make our future release process and testing way smoother.
>
> J.
>