You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@buildstream.apache.org by Tomaz Canabrava <tc...@kde.org> on 2020/09/16 13:20:23 UTC

New Reproducibility test script on for buildstream

Hello, I'd like to invite you to look at the new reproducibility test script
we developed for freedesktop-sdk, I believe that reproducibility testing is
a important feature on any build tool and strangely, it's something that is
missing in all of them. even modern build toolkits like Meson, CMake and
Bazel have no direct way to trigger a build to check reproducibility for
the same architecture.

Buildstream also misses that, and the only way to implement is to bypass
buildstream and hack around the tool. Currently it's not trivial to
implement
this, at all.

You also need to force a build to ignore the remote artifact cache, but
not the sources - as downloading from the remote cache twice and comparing
the results is not actually testing reproducibility.
We have to deal with two things on the connection point of view: we need
to be able to block connection to the remote artifact cache, but at the
same time we do want to download the sources, so that we can download from
the remote cache once, copy the local cache to folder A, build locally
folder B with the downloaded sources, and compare both.

We need a way to disable internet completely for certain actions, but
that's not trivial. The initial script used unshare, but we found out
that this approach created more issues, then we went to a LD_PRELOAD
approach with a library that forbid internet execution.There should be a
better approach for this.

I would like a round of review on the script, as the current idea is to
move that to buildstream as a separate tool. I might missed something (well,
I probably missed something, this is the first time I touch something so
deep in buildstream code).

Currently it will just work on freedesktop-sdk as it requires the LD_PRELOAD
hack, but the idea is to move this to bst in the future, when bst supports
everything that's needed for this.

The script lives currently here:
https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/blob/master/utils/test_reproducibility.py