You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@apache.org by Jarek Potiuk <ja...@potiuk.com> on 2024/01/14 19:06:26 UTC
Reproducible builds [Airflow] -> done
Hey everyone,
I just wanted to share a little accomplishment I (mostly) implemented in
Airflow - I just merged the last PR to get fully reproducible builds for
all thePython artifacts we produce and publish in downloads.apache.org
(python whl, sdist packages, source tarballs).
All our 90 or so artifacts are now fully reproducible and we check
reproducibility of them as a mandatory step of PMC verification when voting
the releases. Initially I thought it's not THAT needed for us in the Python
world, but I got the "let's be reproducible" bug implanted at the
"reproducible builds" talk at the ApacheCon in Halifax by Hervé Boutemy and
it stuck - until I got it completed.
And yeah. It simplified quite a lot our artifact verification and made our
process much more robust.
Arnout just created this page
https://cwiki.apache.org/confluence/display/SECURITY/Reproducible+Builds
where we might gather some notes and guidelines around reproducibility -
and I added some Python notes and links to our small snippets of making the
packages "nicer" reproducible. For example we have now pretty accurate -
yet reproducible - dates in the packages as we store source-date-epoch in
our repository and bump it automatically as part of our release preparation
process. Might be a good idea if others keep their notes there as well
to share experiences.
As a side note (and maybe sparking a bit of a Java/Maven vs. Python
battle), I used to have a bit of an inferiority complex when comparing the
build system state in Python - where we did not have so good standards and
tooling was proliferated and complex. But as part of preparation there I
had to basically move Airflow to the modern packaging world of Python (we
accumulated quite some tech debt and had custom solutions there) but I
learned that with the approach of separate build frontend and build
backends, based on multiple PEP-standards, Python quite leapfrogged the
Java world IMHO. It's a bit of a marvel of what the Python Packaging team
accomplished in the last few years when it comes to standard adoption and
tooling. I am honestly quite impressed with it, and it's just the beginning
to be honest. And it makes our contributor's life so much easier - where
they can stick to whatever frontend they like (Hatch, poetry, flit, pip,
....) and it seamlessly works with a nicely integrated and customizable
build backend of our choice (hatchling in this case).
Looking forward to other stories there in the future.
J.
AW: Reproducible builds [Airflow] -> done
Posted by Christofer Dutz <ch...@c-ware.de>.
In PLC4X we’re also working on enabling fully reproducible builds.
As PLC4X contains Java, C, C#, Go, Python, … it was a bit of a challenge, but I think we’ll cross the finishing lines with our upcoming release.
We basically gave up on the idea of configuring our tooling to produce the same output on every Java Version, Operating-system and CPU architecture,
we simply defined a reference platform in form of a Dockerfile and use docker-compose to “ship the reference build machine”.
So the idea is to execute all release operations in this reference build machine and to also use it to validate releases.
The building and staging of releases has already been finished, still need to finish some of the missing parts, but hopeful that we’ll reach the final goal soon.
Chris
Von: Gary Gregory <ga...@gmail.com>
Datum: Sonntag, 14. Januar 2024 um 20:28
An: security-discuss@community.apache.org <se...@community.apache.org>
Cc: builds <bu...@apache.org>
Betreff: Re: Reproducible builds [Airflow] -> done
Congratulations Jarkek :-)
I was glad to see a reference to
https://maven.apache.org/guides/mini/guide-reproducible-builds.html
I hope we can keep that Maven page up-to-date with whatever comes up so we
only have to look in one place ; -)
Gary
On Sun, Jan 14, 2024, 2:06 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hey everyone,
>
> I just wanted to share a little accomplishment I (mostly) implemented in
> Airflow - I just merged the last PR to get fully reproducible builds for
> all thePython artifacts we produce and publish in downloads.apache.org
> (python whl, sdist packages, source tarballs).
>
> All our 90 or so artifacts are now fully reproducible and we check
> reproducibility of them as a mandatory step of PMC verification when voting
> the releases. Initially I thought it's not THAT needed for us in the Python
> world, but I got the "let's be reproducible" bug implanted at the
> "reproducible builds" talk at the ApacheCon in Halifax by Hervé Boutemy and
> it stuck - until I got it completed.
>
> And yeah. It simplified quite a lot our artifact verification and made our
> process much more robust.
>
> Arnout just created this page
> https://cwiki.apache.org/confluence/display/SECURITY/Reproducible+Builds
> where we might gather some notes and guidelines around reproducibility -
> and I added some Python notes and links to our small snippets of making the
> packages "nicer" reproducible. For example we have now pretty accurate -
> yet reproducible - dates in the packages as we store source-date-epoch in
> our repository and bump it automatically as part of our release preparation
> process. Might be a good idea if others keep their notes there as well
> to share experiences.
>
> As a side note (and maybe sparking a bit of a Java/Maven vs. Python
> battle), I used to have a bit of an inferiority complex when comparing the
> build system state in Python - where we did not have so good standards and
> tooling was proliferated and complex. But as part of preparation there I
> had to basically move Airflow to the modern packaging world of Python (we
> accumulated quite some tech debt and had custom solutions there) but I
> learned that with the approach of separate build frontend and build
> backends, based on multiple PEP-standards, Python quite leapfrogged the
> Java world IMHO. It's a bit of a marvel of what the Python Packaging team
> accomplished in the last few years when it comes to standard adoption and
> tooling. I am honestly quite impressed with it, and it's just the beginning
> to be honest. And it makes our contributor's life so much easier - where
> they can stick to whatever frontend they like (Hatch, poetry, flit, pip,
> ....) and it seamlessly works with a nicely integrated and customizable
> build backend of our choice (hatchling in this case).
>
> Looking forward to other stories there in the future.
>
> J.
>
AW: Reproducible builds [Airflow] -> done
Posted by Christofer Dutz <ch...@c-ware.de>.
In PLC4X we’re also working on enabling fully reproducible builds.
As PLC4X contains Java, C, C#, Go, Python, … it was a bit of a challenge, but I think we’ll cross the finishing lines with our upcoming release.
We basically gave up on the idea of configuring our tooling to produce the same output on every Java Version, Operating-system and CPU architecture,
we simply defined a reference platform in form of a Dockerfile and use docker-compose to “ship the reference build machine”.
So the idea is to execute all release operations in this reference build machine and to also use it to validate releases.
The building and staging of releases has already been finished, still need to finish some of the missing parts, but hopeful that we’ll reach the final goal soon.
Chris
Von: Gary Gregory <ga...@gmail.com>
Datum: Sonntag, 14. Januar 2024 um 20:28
An: security-discuss@community.apache.org <se...@community.apache.org>
Cc: builds <bu...@apache.org>
Betreff: Re: Reproducible builds [Airflow] -> done
Congratulations Jarkek :-)
I was glad to see a reference to
https://maven.apache.org/guides/mini/guide-reproducible-builds.html
I hope we can keep that Maven page up-to-date with whatever comes up so we
only have to look in one place ; -)
Gary
On Sun, Jan 14, 2024, 2:06 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hey everyone,
>
> I just wanted to share a little accomplishment I (mostly) implemented in
> Airflow - I just merged the last PR to get fully reproducible builds for
> all thePython artifacts we produce and publish in downloads.apache.org
> (python whl, sdist packages, source tarballs).
>
> All our 90 or so artifacts are now fully reproducible and we check
> reproducibility of them as a mandatory step of PMC verification when voting
> the releases. Initially I thought it's not THAT needed for us in the Python
> world, but I got the "let's be reproducible" bug implanted at the
> "reproducible builds" talk at the ApacheCon in Halifax by Hervé Boutemy and
> it stuck - until I got it completed.
>
> And yeah. It simplified quite a lot our artifact verification and made our
> process much more robust.
>
> Arnout just created this page
> https://cwiki.apache.org/confluence/display/SECURITY/Reproducible+Builds
> where we might gather some notes and guidelines around reproducibility -
> and I added some Python notes and links to our small snippets of making the
> packages "nicer" reproducible. For example we have now pretty accurate -
> yet reproducible - dates in the packages as we store source-date-epoch in
> our repository and bump it automatically as part of our release preparation
> process. Might be a good idea if others keep their notes there as well
> to share experiences.
>
> As a side note (and maybe sparking a bit of a Java/Maven vs. Python
> battle), I used to have a bit of an inferiority complex when comparing the
> build system state in Python - where we did not have so good standards and
> tooling was proliferated and complex. But as part of preparation there I
> had to basically move Airflow to the modern packaging world of Python (we
> accumulated quite some tech debt and had custom solutions there) but I
> learned that with the approach of separate build frontend and build
> backends, based on multiple PEP-standards, Python quite leapfrogged the
> Java world IMHO. It's a bit of a marvel of what the Python Packaging team
> accomplished in the last few years when it comes to standard adoption and
> tooling. I am honestly quite impressed with it, and it's just the beginning
> to be honest. And it makes our contributor's life so much easier - where
> they can stick to whatever frontend they like (Hatch, poetry, flit, pip,
> ....) and it seamlessly works with a nicely integrated and customizable
> build backend of our choice (hatchling in this case).
>
> Looking forward to other stories there in the future.
>
> J.
>
Re: Reproducible builds [Airflow] -> done
Posted by Gary Gregory <ga...@gmail.com>.
Congratulations Jarkek :-)
I was glad to see a reference to
https://maven.apache.org/guides/mini/guide-reproducible-builds.html
I hope we can keep that Maven page up-to-date with whatever comes up so we
only have to look in one place ; -)
Gary
On Sun, Jan 14, 2024, 2:06 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hey everyone,
>
> I just wanted to share a little accomplishment I (mostly) implemented in
> Airflow - I just merged the last PR to get fully reproducible builds for
> all thePython artifacts we produce and publish in downloads.apache.org
> (python whl, sdist packages, source tarballs).
>
> All our 90 or so artifacts are now fully reproducible and we check
> reproducibility of them as a mandatory step of PMC verification when voting
> the releases. Initially I thought it's not THAT needed for us in the Python
> world, but I got the "let's be reproducible" bug implanted at the
> "reproducible builds" talk at the ApacheCon in Halifax by Hervé Boutemy and
> it stuck - until I got it completed.
>
> And yeah. It simplified quite a lot our artifact verification and made our
> process much more robust.
>
> Arnout just created this page
> https://cwiki.apache.org/confluence/display/SECURITY/Reproducible+Builds
> where we might gather some notes and guidelines around reproducibility -
> and I added some Python notes and links to our small snippets of making the
> packages "nicer" reproducible. For example we have now pretty accurate -
> yet reproducible - dates in the packages as we store source-date-epoch in
> our repository and bump it automatically as part of our release preparation
> process. Might be a good idea if others keep their notes there as well
> to share experiences.
>
> As a side note (and maybe sparking a bit of a Java/Maven vs. Python
> battle), I used to have a bit of an inferiority complex when comparing the
> build system state in Python - where we did not have so good standards and
> tooling was proliferated and complex. But as part of preparation there I
> had to basically move Airflow to the modern packaging world of Python (we
> accumulated quite some tech debt and had custom solutions there) but I
> learned that with the approach of separate build frontend and build
> backends, based on multiple PEP-standards, Python quite leapfrogged the
> Java world IMHO. It's a bit of a marvel of what the Python Packaging team
> accomplished in the last few years when it comes to standard adoption and
> tooling. I am honestly quite impressed with it, and it's just the beginning
> to be honest. And it makes our contributor's life so much easier - where
> they can stick to whatever frontend they like (Hatch, poetry, flit, pip,
> ....) and it seamlessly works with a nicely integrated and customizable
> build backend of our choice (hatchling in this case).
>
> Looking forward to other stories there in the future.
>
> J.
>
Re: Reproducible builds [Airflow] -> done
Posted by Gary Gregory <ga...@gmail.com>.
Congratulations Jarkek :-)
I was glad to see a reference to
https://maven.apache.org/guides/mini/guide-reproducible-builds.html
I hope we can keep that Maven page up-to-date with whatever comes up so we
only have to look in one place ; -)
Gary
On Sun, Jan 14, 2024, 2:06 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hey everyone,
>
> I just wanted to share a little accomplishment I (mostly) implemented in
> Airflow - I just merged the last PR to get fully reproducible builds for
> all thePython artifacts we produce and publish in downloads.apache.org
> (python whl, sdist packages, source tarballs).
>
> All our 90 or so artifacts are now fully reproducible and we check
> reproducibility of them as a mandatory step of PMC verification when voting
> the releases. Initially I thought it's not THAT needed for us in the Python
> world, but I got the "let's be reproducible" bug implanted at the
> "reproducible builds" talk at the ApacheCon in Halifax by Hervé Boutemy and
> it stuck - until I got it completed.
>
> And yeah. It simplified quite a lot our artifact verification and made our
> process much more robust.
>
> Arnout just created this page
> https://cwiki.apache.org/confluence/display/SECURITY/Reproducible+Builds
> where we might gather some notes and guidelines around reproducibility -
> and I added some Python notes and links to our small snippets of making the
> packages "nicer" reproducible. For example we have now pretty accurate -
> yet reproducible - dates in the packages as we store source-date-epoch in
> our repository and bump it automatically as part of our release preparation
> process. Might be a good idea if others keep their notes there as well
> to share experiences.
>
> As a side note (and maybe sparking a bit of a Java/Maven vs. Python
> battle), I used to have a bit of an inferiority complex when comparing the
> build system state in Python - where we did not have so good standards and
> tooling was proliferated and complex. But as part of preparation there I
> had to basically move Airflow to the modern packaging world of Python (we
> accumulated quite some tech debt and had custom solutions there) but I
> learned that with the approach of separate build frontend and build
> backends, based on multiple PEP-standards, Python quite leapfrogged the
> Java world IMHO. It's a bit of a marvel of what the Python Packaging team
> accomplished in the last few years when it comes to standard adoption and
> tooling. I am honestly quite impressed with it, and it's just the beginning
> to be honest. And it makes our contributor's life so much easier - where
> they can stick to whatever frontend they like (Hatch, poetry, flit, pip,
> ....) and it seamlessly works with a nicely integrated and customizable
> build backend of our choice (hatchling in this case).
>
> Looking forward to other stories there in the future.
>
> J.
>
Re: Reproducible builds [Airflow] -> done
Posted by Jarek Potiuk <ja...@potiuk.com>.
@ Chris:
> We basically gave up on the idea of configuring our tooling to produce
the same output on every Java Version, Operating-system and CPU
architecture,
we simply defined a reference platform in form of a Dockerfile and use
docker-compose to “ship the reference build machine”.
Yep - we also do it for some parts of the build - especially in CI, where
we want to run user's code coming from PR in an isolated environment. But
in many cases we do not have to do it, as having just a fixed set of
"Python" deps is enough - and modern tooling for Python installs such
"isolated build environment" for regular builds, so this is happens pretty
much behind-the-scenes.
@ Piotr
> If the builds are reproducible and PMCs are required to check it, what
do you think about proving reproducibility to the outer world by
asking PMCs to cosign the published artifacts?
Why not. Main problem with multiple signatures was that it was difficult to
confirm provenance of the binary to be signed - but if it is binary
identical - why not. I think also the future "INFRA artifact platform"
should allow such signing (especially when we will use sigstore).
One comment:
I updated
https://cwiki.apache.org/confluence/display/SECURITY/Reproducible+Builds
with some notes about umask (learning from Airflow's reproducibility)
BTW. I am just about to merge a change where i also introduce
reproducibility to preparing our Helm Chart packages.
On Mon, Jan 15, 2024 at 8:29 AM Piotr P. Karwasz <pi...@gmail.com>
wrote:
> Hi Jarek,
>
> On Sun, 14 Jan 2024 at 20:06, Jarek Potiuk <ja...@potiuk.com> wrote:
> > I just wanted to share a little accomplishment I (mostly) implemented in
> > Airflow - I just merged the last PR to get fully reproducible builds for
> > all thePython artifacts we produce and publish in downloads.apache.org
> > (python whl, sdist packages, source tarballs).
> >
> > All our 90 or so artifacts are now fully reproducible and we check
> > reproducibility of them as a mandatory step of PMC verification when
> voting
> > the releases. Initially I thought it's not THAT needed for us in the
> Python
> > world, but I got the "let's be reproducible" bug implanted at the
> > "reproducible builds" talk at the ApacheCon in Halifax by Hervé Boutemy
> and
> > it stuck - until I got it completed.
>
> Congratulations!
>
> If the builds are reproducible and PMCs are required to check it, what
> do you think about proving reproducibility to the outer world by
> asking PMCs to cosign the published artifacts?
>
> This additional step would certainly require some changes in the tools
> we use to publish artifacts. For example in the Nexus repo it is
> impossible (AFAIK) to modify the signatures after the staging repo has
> been closed. I have opened an INFRA ticket to see if that can be
> changed:
>
> https://issues.apache.org/jira/browse/INFRA-25381
>
> Piotr
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: security-discuss-unsubscribe@community.apache.org
> For additional commands, e-mail:
> security-discuss-help@community.apache.org
>
>
Re: Reproducible builds [Airflow] -> done
Posted by "Piotr P. Karwasz" <pi...@gmail.com>.
Hi Jarek,
On Sun, 14 Jan 2024 at 20:06, Jarek Potiuk <ja...@potiuk.com> wrote:
> I just wanted to share a little accomplishment I (mostly) implemented in
> Airflow - I just merged the last PR to get fully reproducible builds for
> all thePython artifacts we produce and publish in downloads.apache.org
> (python whl, sdist packages, source tarballs).
>
> All our 90 or so artifacts are now fully reproducible and we check
> reproducibility of them as a mandatory step of PMC verification when voting
> the releases. Initially I thought it's not THAT needed for us in the Python
> world, but I got the "let's be reproducible" bug implanted at the
> "reproducible builds" talk at the ApacheCon in Halifax by Hervé Boutemy and
> it stuck - until I got it completed.
Congratulations!
If the builds are reproducible and PMCs are required to check it, what
do you think about proving reproducibility to the outer world by
asking PMCs to cosign the published artifacts?
This additional step would certainly require some changes in the tools
we use to publish artifacts. For example in the Nexus repo it is
impossible (AFAIK) to modify the signatures after the staging repo has
been closed. I have opened an INFRA ticket to see if that can be
changed:
https://issues.apache.org/jira/browse/INFRA-25381
Piotr
---------------------------------------------------------------------
To unsubscribe, e-mail: security-discuss-unsubscribe@community.apache.org
For additional commands, e-mail: security-discuss-help@community.apache.org