You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by Mark Thomas <ma...@apache.org> on 2022/03/22 17:12:29 UTC
Repeatable builds
Hi all,
I wanted to provide a status update on this.
The bnd issue [1] has been fixed and will be included in the next
release. We can switch to the latest bnd snapshot to pick up that fix in
the meantime.
I have tested detached signatures with JSign and they do work across
platforms. I was able to build 10.1.0-M13-dev on Linux (generating the
signature) and then repeat the build on Windows this time inserting the
signature and the uninstaller was correctly digitally signed.
There are, however, a couple of issues.
The JSign Ant task that adds the detached signature doesn't close the
signed file. This causes problems for Ant. I've opened a JSign issue [2]
for this. I have a locally build version with a hack that fixes the
issue so I can continue testing.
The zips generated by the Javadoc task don't fix the timestamps of the
files within the zips. The breaks repeatability for the full-docs
package and for the Windows installer.
If it were just the full docs package, I don't think I'd worry too much
about the Javadoc issue but the Windows installer is more of a problem.
Therefore, I plan to work on a custom Ant task that will fix these zip
files by setting the timestamps on the compressed files to be the same
as the timestamp used by the rest of the build.
Finally, I have some minor modifications to buidl.xml that will enable
repeatable builds. Once everything is in place the build process will
look like:
- prepare for tagging as currently (update version in
build.properties.default and edit changelog)
- run a release build to generate the detached signatures
- tag including:
- modified build.properties.default
- modified chnagelog.xml
- 2x .sig files added to res/install-win
- anyone can build a release from the tag and will get a build that
includes a signed Windows installer
I have confirmed the builds are repeatable across Linux and Windows. I
need to work on how much variation is permitted for Ant versions and JRE
versions.
Mark
[1] https://github.com/bndtools/bnd/issues/5183
[2] https://github.com/ebourg/jsign/issues/117
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Repeatable builds
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Mark,
On 3/23/22 12:18, Mark Thomas wrote:
> On 23/03/2022 16:11, Christopher Schultz wrote:
>> ??
>> $ ant javadoc
>>
>> Dumps everything into output/dist/webapps/docs/, no ZIP files. The
>> "javadoc" target executes <javadoc> tasks and that's it.
>>
>> What am I missing?
>
> Look in the individual Javadoc directories. Each one has (I think) 3 zip
> files that contain various indexes.
$ ant clean javascript
[...]
$ find output/dist/webapps/docs/ -name "*.zip" -o -name "*.jar"
$
I don't see those. Maybe it's my Java version?
$ java -version
openjdk version "16.0.2" 2021-07-20
OpenJDK Runtime Environment (build 16.0.2+7-67)
OpenJDK 64-Bit Server VM (build 16.0.2+7-67, mixed mode, sharing)
$ head output/dist/webapps/docs/servletapi/index.html
<!DOCTYPE HTML>
<!-- NewPage -->
<html lang="en">
<head>
<!-- Generated by javadoc (16) -->
<title>Overview (Servlet {servlet.spec.version} API Documentation -
Apache Tomcat 10.1.0-M13-dev)</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="description" content="package index">
<meta name="generator" content="javadoc/PackageIndexWriter">
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="Style">
<link rel="stylesheet" type="text/css"
href="script-dir/jquery-ui.min.css" title="Style">
<link rel="stylesheet" type="text/css" href="jquery-ui.overrides.css"
title="Style">
<script type="text/javascript" src="script.js"></script>
<script type="text/javascript"
src="script-dir/jquery-3.5.1.min.js"></script>
<script type="text/javascript" src="script-dir/jquery-ui.min.js"></script>
</head>
So it's definitely using the Java 16 javadoc engine.
>>>> Hopefully, anything Java 11 or later with e.g. "-actlike 1.7" will
>>>> produce binary-identical artifacts. If not, merely stating that a
>>>> particular build was done with some exact JDK version should be enough.
>>>
>>> Hopefully we'll be able to find a way to be reasonably flexible on
>>> Java versions but we'll have to see.
>>
>> As long as some other party who wishes to confirm the release hasn't
>> been tampered-with will be able to use the same toolchain, assuming
>> it's documented properly. This is what non-Java projects do for
>> repeatability. You can't get the same bytes when using gcc versus
>> clang, for example. Same with various versions of those things.
>
> Same tool-chain should definitely work. But I'd still like to see how
> much flexibility - if any - we have in versions of Java and Ant.
>
>>> Sounds good. Anything that makes the release process easier is a good
>>> thing (although compared to what it used to be like the current
>>> process is a breeze).
>>
>> :)
>>
>> Okay, I'll try to do those in advance of the next releases.
>
> Cool.
I'm having some trouble getting <tstamp> to give me
current-timestamp-in-seconds. I can get the ISO date just fine.
This SO answer [https://stackoverflow.com/a/25535927/276232] doesn't
seems to work out--of-the-box; I get an error about not being able to
initialize the javascript engine.
I tried the obvious pattern="sssssssssss" but I just got a bunch of
zeros followed by the seconds-in-the-current-minute. :/
I have it building the following build-release.properties file:
=== CUT ===
# These ant.tstamp properties specify the build-timestamp for repeatable
builds.
ant.tstamp.now=000000000000000041
ant.tstamp.now.iso=2022-03-23T16:48:41Z
# This is the ASF user who is the Release Manager for this release.
release.asfusername=schultz
# Set the version-suffix to "" (empty string) as this is not a
development release.
version.suffix=
# Release build environment information:
release.java.version=16.0.2
release.java.vm.name=OpenJDK 64-Bit Server VM
release.java.vm.version=16.0.2+7-67
release.os=x86_64 Mac OS X 11.5
release.default.file.encoding=UTF-8
=== CUT ===
So... almost everything I want :)
Any ideas for the "time in seconds" thing? (I may be able to get ant
working with javascript, but it requires a bunch of dependencies
including GraalVM, etc. - !!!).
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Repeatable builds
Posted by Mark Thomas <ma...@apache.org>.
On 23/03/2022 16:11, Christopher Schultz wrote:
> Mark,
>
> On 3/22/22 15:30, Mark Thomas wrote:
>> On 22/03/2022 19:06, Christopher Schultz wrote:
>>> On 3/22/22 13:12, Mark Thomas wrote:
>>
>> <snip/>
>>
>>>> The JSign Ant task that adds the detached signature doesn't close
>>>> the signed file. This causes problems for Ant. I've opened a JSign
>>>> issue [2] for this. I have a locally build version with a hack that
>>>> fixes the issue so I can continue testing.
>>>
>>> Do you mean it doesn't call OutputStream.close()? Shouldn't that be
>>> cleaned up when the process exits?
>>
>> Effectively, yes. The problem is that the Ant task runs in process so
>> the file is still open when the next Ant target tries to work with the
>> file. You won't notice the issue running JSign from the command line.
>
> Can we run the jsign task with fork="true" or is there too much
> information in the ant process's memory that would need to be
> insecurely-sent to the jsign process to be prudent?
I don't think it supports it. If it did, that would be another workaround.
>> No. The Javadoc task generates a file then zips it. We need to change
>> the timestamp of the file(s) inside the created zip.
>
> ??
> $ ant javadoc
>
> Dumps everything into output/dist/webapps/docs/, no ZIP files. The
> "javadoc" target executes <javadoc> tasks and that's it.
>
> What am I missing?
Look in the individual Javadoc directories. Each one has (I think) 3 zip
files that contain various indexes.
>>> Hopefully, anything Java 11 or later with e.g. "-actlike 1.7" will
>>> produce binary-identical artifacts. If not, merely stating that a
>>> particular build was done with some exact JDK version should be enough.
>>
>> Hopefully we'll be able to find a way to be reasonably flexible on
>> Java versions but we'll have to see.
>
> As long as some other party who wishes to confirm the release hasn't
> been tampered-with will be able to use the same toolchain, assuming it's
> documented properly. This is what non-Java projects do for
> repeatability. You can't get the same bytes when using gcc versus clang,
> for example. Same with various versions of those things.
Same tool-chain should definitely work. But I'd still like to see how
much flexibility - if any - we have in versions of Java and Ant.
>> Sounds good. Anything that makes the release process easier is a good
>> thing (although compared to what it used to be like the current
>> process is a breeze).
>
> :)
>
> Okay, I'll try to do those in advance of the next releases.
Cool.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Repeatable builds
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Mark,
On 3/22/22 15:30, Mark Thomas wrote:
> On 22/03/2022 19:06, Christopher Schultz wrote:
>> On 3/22/22 13:12, Mark Thomas wrote:
>
> <snip/>
>
>>> The JSign Ant task that adds the detached signature doesn't close the
>>> signed file. This causes problems for Ant. I've opened a JSign issue
>>> [2] for this. I have a locally build version with a hack that fixes
>>> the issue so I can continue testing.
>>
>> Do you mean it doesn't call OutputStream.close()? Shouldn't that be
>> cleaned up when the process exits?
>
> Effectively, yes. The problem is that the Ant task runs in process so
> the file is still open when the next Ant target tries to work with the
> file. You won't notice the issue running JSign from the command line.
Can we run the jsign task with fork="true" or is there too much
information in the ant process's memory that would need to be
insecurely-sent to the jsign process to be prudent?
>>> If it were just the full docs package, I don't think I'd worry too
>>> much about the Javadoc issue but the Windows installer is more of a
>>> problem. Therefore, I plan to work on a custom Ant task that will fix
>>> these zip files by setting the timestamps on the compressed files to
>>> be the same as the timestamp used by the rest of the build.
>>
>> Can <touch> not be used to do this?
>
> No. The Javadoc task generates a file then zips it. We need to change
> the timestamp of the file(s) inside the created zip.
??
$ ant javadoc
Dumps everything into output/dist/webapps/docs/, no ZIP files. The
"javadoc" target executes <javadoc> tasks and that's it.
What am I missing?
>>> - anyone can build a release from the tag and will get a build that
>>> includes a signed Windows installer
>>
>> Oh, that last part is clever, which of course is the whole point of the
>> detached signatures.
>
> When it is finally working it is going to be really nice. We'll
> effectively have a way for anyone to build from source and confirm that
> they have exactly the same bits as the official release. If provides a
> great way to demonstrate that the RM hasn't tampered with the binaries.
+1
>> Hopefully, anything Java 11 or later with e.g. "-actlike 1.7" will
>> produce binary-identical artifacts. If not, merely stating that a
>> particular build was done with some exact JDK version should be enough.
>
> Hopefully we'll be able to find a way to be reasonably flexible on Java
> versions but we'll have to see.
As long as some other party who wishes to confirm the release hasn't
been tampered-with will be able to use the same toolchain, assuming it's
documented properly. This is what non-Java projects do for
repeatability. You can't get the same bytes when using gcc versus clang,
for example. Same with various versions of those things.
>> The release-build should probably auto-generate a report of the
>> build-chain used to build the release.
>
> Nice idea.
>
>> Other potential improvements to the build process I've been
>> considering making:
>>
>> 1. $ ant prep-release
>> Generates a new build.properties file, build-release.properties which
>> contains a few useful items:
>> ant.tstamp.now=[timestamp to be used for repeatable build]
>> version.suffix= (intentionally blank)
>> release.asfuser=[username of the user rolling the release]
>> release.jdk=[details of build environment, perhaps additional items]
>> release.os=[ibid]
>>
>> Then modify the build script to include build-release.properties
>> before any others and ignore the file if it's missing.
>>
>> Also this will generate res/maven/build-release.properties which will
>> include:
>> asf.ldap.username
>> maven.asf.release.deploy.version
>> gpg.exec
>>
>> There's no reason these items can't be computed from the outer build
>> process and used inside there. It eliminates a few steps for the
>> person doing the release.
>>
>> 2. Re-name res/maven/mvn-pub.xml -> build.xml
>>
>> This saves unnecessary typing during the release. I'll probably
>> actually move the file, then create a new one in its place pointing to
>> the new one, in case some other downstream process needs that file to
>> be in-place.
>
> I doubt that will be necessary. The file is very specific to how the ASF
> is set up.
>
>> 3. Read build-release.properties in build.xml (nee mvn-pub.xml) as
>> well as build.properties and build.properties.default.
>> build.properties should be almost unnecessary at this point.
>>
>> 4. Document the mail build <target>s in build.xml (nee mvn-pub.xml)
>>
>> I'm sure there are other things I've been thinking of. These are the
>> ones I have in the front of my mind.
>
> Sounds good. Anything that makes the release process easier is a good
> thing (although compared to what it used to be like the current process
> is a breeze).
:)
Okay, I'll try to do those in advance of the next releases.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Repeatable builds
Posted by Mark Thomas <ma...@apache.org>.
On 22/03/2022 19:06, Christopher Schultz wrote:
> On 3/22/22 13:12, Mark Thomas wrote:
<snip/>
>> The JSign Ant task that adds the detached signature doesn't close the
>> signed file. This causes problems for Ant. I've opened a JSign issue
>> [2] for this. I have a locally build version with a hack that fixes
>> the issue so I can continue testing.
>
> Do you mean it doesn't call OutputStream.close()? Shouldn't that be
> cleaned up when the process exits?
Effectively, yes. The problem is that the Ant task runs in process so
the file is still open when the next Ant target tries to work with the
file. You won't notice the issue running JSign from the command line.
<snip/>
>> If it were just the full docs package, I don't think I'd worry too
>> much about the Javadoc issue but the Windows installer is more of a
>> problem. Therefore, I plan to work on a custom Ant task that will fix
>> these zip files by setting the timestamps on the compressed files to
>> be the same as the timestamp used by the rest of the build.
>
> Can <touch> not be used to do this?
No. The Javadoc task generates a file then zips it. We need to change
the timestamp of the file(s) inside the created zip.
<snip/>
>> - anyone can build a release from the tag and will get a build that
>> includes a signed Windows installer
>
> Oh, that last part is clever, which of course is the whole point of the
> detached signatures.
When it is finally working it is going to be really nice. We'll
effectively have a way for anyone to build from source and confirm that
they have exactly the same bits as the official release. If provides a
great way to demonstrate that the RM hasn't tampered with the binaries.
<snip/>
> Hopefully, anything Java 11 or later with e.g. "-actlike 1.7" will
> produce binary-identical artifacts. If not, merely stating that a
> particular build was done with some exact JDK version should be enough.
Hopefully we'll be able to find a way to be reasonably flexible on Java
versions but we'll have to see.
<snip/>
> The release-build should probably auto-generate a report of the
> build-chain used to build the release.
Nice idea.
> Other potential improvements to the build process I've been considering
> making:
>
> 1. $ ant prep-release
> Generates a new build.properties file, build-release.properties which
> contains a few useful items:
> ant.tstamp.now=[timestamp to be used for repeatable build]
> version.suffix= (intentionally blank)
> release.asfuser=[username of the user rolling the release]
> release.jdk=[details of build environment, perhaps additional items]
> release.os=[ibid]
>
> Then modify the build script to include build-release.properties before
> any others and ignore the file if it's missing.
>
> Also this will generate res/maven/build-release.properties which will
> include:
> asf.ldap.username
> maven.asf.release.deploy.version
> gpg.exec
>
> There's no reason these items can't be computed from the outer build
> process and used inside there. It eliminates a few steps for the person
> doing the release.
>
> 2. Re-name res/maven/mvn-pub.xml -> build.xml
>
> This saves unnecessary typing during the release. I'll probably actually
> move the file, then create a new one in its place pointing to the new
> one, in case some other downstream process needs that file to be in-place.
I doubt that will be necessary. The file is very specific to how the ASF
is set up.
> 3. Read build-release.properties in build.xml (nee mvn-pub.xml) as well
> as build.properties and build.properties.default. build.properties
> should be almost unnecessary at this point.
>
> 4. Document the mail build <target>s in build.xml (nee mvn-pub.xml)
>
> I'm sure there are other things I've been thinking of. These are the
> ones I have in the front of my mind.
Sounds good. Anything that makes the release process easier is a good
thing (although compared to what it used to be like the current process
is a breeze).
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Repeatable builds
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Mark,
On 3/22/22 13:12, Mark Thomas wrote:
> Hi all,
>
> I wanted to provide a status update on this.
>
> The bnd issue [1] has been fixed and will be included in the next
> release. We can switch to the latest bnd snapshot to pick up that fix in
> the meantime.
>
> I have tested detached signatures with JSign and they do work across
> platforms. I was able to build 10.1.0-M13-dev on Linux (generating the
> signature) and then repeat the build on Windows this time inserting the
> signature and the uninstaller was correctly digitally signed.
>
> There are, however, a couple of issues.
>
> The JSign Ant task that adds the detached signature doesn't close the
> signed file. This causes problems for Ant. I've opened a JSign issue [2]
> for this. I have a locally build version with a hack that fixes the
> issue so I can continue testing.
Do you mean it doesn't call OutputStream.close()? Shouldn't that be
cleaned up when the process exits?
> The zips generated by the Javadoc task don't fix the timestamps of the
> files within the zips. The breaks repeatability for the full-docs
> package and for the Windows installer.
>
> If it were just the full docs package, I don't think I'd worry too much
> about the Javadoc issue but the Windows installer is more of a problem.
> Therefore, I plan to work on a custom Ant task that will fix these zip
> files by setting the timestamps on the compressed files to be the same
> as the timestamp used by the rest of the build.
Can <touch> not be used to do this?
> Finally, I have some minor modifications to buidl.xml that will enable
> repeatable builds. Once everything is in place the build process will
> look like:
>
> - prepare for tagging as currently (update version in
> build.properties.default and edit changelog)
> - run a release build to generate the detached signatures
> - tag including:
> - modified build.properties.default
> - modified chnagelog.xml
> - 2x .sig files added to res/install-win
> - anyone can build a release from the tag and will get a build that
> includes a signed Windows installer
Oh, that last part is clever, which of course is the whole point of the
detached signatures.
> I have confirmed the builds are repeatable across Linux and Windows. I
> need to work on how much variation is permitted for Ant versions and JRE
> versions.
Hopefully, anything Java 11 or later with e.g. "-actlike 1.7" will
produce binary-identical artifacts. If not, merely stating that a
particular build was done with some exact JDK version should be enough.
The release-build should probably auto-generate a report of the
build-chain used to build the release.
Other potential improvements to the build process I've been considering
making:
1. $ ant prep-release
Generates a new build.properties file, build-release.properties which
contains a few useful items:
ant.tstamp.now=[timestamp to be used for repeatable build]
version.suffix= (intentionally blank)
release.asfuser=[username of the user rolling the release]
release.jdk=[details of build environment, perhaps additional items]
release.os=[ibid]
Then modify the build script to include build-release.properties before
any others and ignore the file if it's missing.
Also this will generate res/maven/build-release.properties which will
include:
asf.ldap.username
maven.asf.release.deploy.version
gpg.exec
There's no reason these items can't be computed from the outer build
process and used inside there. It eliminates a few steps for the person
doing the release.
2. Re-name res/maven/mvn-pub.xml -> build.xml
This saves unnecessary typing during the release. I'll probably actually
move the file, then create a new one in its place pointing to the new
one, in case some other downstream process needs that file to be in-place.
3. Read build-release.properties in build.xml (nee mvn-pub.xml) as well
as build.properties and build.properties.default. build.properties
should be almost unnecessary at this point.
4. Document the mail build <target>s in build.xml (nee mvn-pub.xml)
I'm sure there are other things I've been thinking of. These are the
ones I have in the front of my mind.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Repeatable builds
Posted by Mark Thomas <ma...@apache.org>.
On 23/03/2022 17:20, Christopher Schultz wrote:
> Mark,
>
> On 3/22/22 13:12, Mark Thomas wrote:
>> Finally, I have some minor modifications to buidl.xml that will enable
>> repeatable builds. Once everything is in place the build process will
>> look like:
>>
>> - prepare for tagging as currently (update version in
>> build.properties.default and edit changelog)
>> - run a release build to generate the detached signatures
>> - tag including:
>> - modified build.properties.default
>> - modified chnagelog.xml
>> - 2x .sig files added to res/install-win
>
> So this means we have to build *before* the tag, right? Or do we tag the
> repo, do the build, and then tag again with some other related name?
>
> (If the former, it's nice because sometimes I forget to do a "test
> build" before the tag and then I have to fix something (e.g. checkstyle)
> to allow the build to proceed, and I need to re-tag.)
Yes, you need to build before the tag to generate the detached
signatures and then do the actual release build so the source archives
have the detached signatures.
In other news, I was wrong about the Javadoc issue affecting the Windows
installer. However, I have fixed the Javadoc issue anyway. I'll be
committing that shortly since I've spent the time to fix it and it is a
nice to have.
I'm currently at the point where the build is 100% repeatable cross
platform (Windows and Linux) if I don't sign the windows installer.
Something isn't quite right when I sign the installer but I need to
figure out what that is - probably configuration error on my part.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Repeatable builds
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Mark,
On 3/22/22 13:12, Mark Thomas wrote:
> Finally, I have some minor modifications to buidl.xml that will enable
> repeatable builds. Once everything is in place the build process will
> look like:
>
> - prepare for tagging as currently (update version in
> build.properties.default and edit changelog)
> - run a release build to generate the detached signatures
> - tag including:
> - modified build.properties.default
> - modified chnagelog.xml
> - 2x .sig files added to res/install-win
So this means we have to build *before* the tag, right? Or do we tag the
repo, do the build, and then tag again with some other related name?
(If the former, it's nice because sometimes I forget to do a "test
build" before the tag and then I have to fix something (e.g. checkstyle)
to allow the build to proceed, and I need to re-tag.)
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org