You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by Mark Thomas <ma...@apache.org> on 2022/05/05 19:28:21 UTC
Repeatable builds update
TL;DR we have platform independent repeatable release builds
And now for the longer version:
Nearly all of the plumbing is in place for this. I have one more commit
to add to handle repeatable signing of the Windows installer but I am
holding that back as we need a couple of other things to fall into place
first.
1. We need BND 6.3.0. This is because 6.3.0 contains a fix to a
repeatability issue in the manifest generation. The 6.3.0 release is in
progress. It is current at RC1 and a final release is expected in a
couple of weeks.
2. We need JSign 4.1. I found a couple of minor issues in JSign earlier
today. I've hacked around them locally to test the repeatable build but
we need a proper release. ebourg is working on proper fixes to replace
my hacks. On past experience, I expect he'll have a release out before
BND does.
Once all of the above is in place our release builds will be repeatable
on Windows and Linux provided that:
- The same version of Ant is used
- The same JDK (vendor and version) is used
The reason for the ANT and JDK version requirements is that the version
numbers get placed in the manifests.
I have tested this with building on Linux and then repeating the build
(using the generated signatures for the installer files) on Windows. The
results are bit for bit identical.
The release process will need a small change. Essentially, the release
manager will make the usual version changes, call 'ant release' once to
generate the detached signature files and then tag with the version
updates and the signature files. Then the release manager calls 'ant
release' again to create the actual release.
At this point anyone can take the tag and generate the same release
binaries from the tag.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Repeatable builds update
Posted by Mark Thomas <ma...@apache.org>.
On 16/05/2022 18:54, Christopher Schultz wrote:
> I had considered using ant.tstamp.now.iso (you can see I had it in the
> patch, actually), but I think it changes the required version of ant for
> building. Since this is expected to be used by Tomcat release-managers
> only, I think that's a reasonable requirement to put on the ant version.
> We may want to put some checks into the build so ensure that if
> ant.tstamp.now.iso is set, you have a version of ant which is high
> enough to respect it and terminate the build if necessary.
>
> WDYT?
Ant 1.10.x requires Java 8 (rather than Java 5). I'm fine with that
being a requirement for anyone building Tomcat.
> I haven't played with it in a while, but I seem to recall that if you
> execute "pre-release" and "release" at the same time (like "ant
> pre-release release"), weird things happen with the timestamp. Also if
> you do "pre-release" after having run "pre-release" you get some
> weirdness. I think. I'll have to go back and look :)
We might want to hide a few targets. I suspect they were made visible
just so they could be tested during development.
> Finally, "clean" should probably remove the build-release.properties file.
That could be problematic. The release target depends on the clean target.
I think we might need another property that triggers generation of the
file if not present. That way, release managers can set that property on
their release systems but it won't get in the way of anyone else using
"ant release".
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Repeatable builds update
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Mark,
On 5/16/22 12:30, Mark Thomas wrote:
> On 06/05/2022 14:27, Christopher Schultz wrote:
>
>> Awesome. I have a small commit to build.xml that drops environmental
>> information (e.g. versions, etc.) to a build-release.properties file,
>> but I'm unable to generate the timestamp in the correct format in
>> order to put it into the file. Here's the patch for it; I'd love it if
>> anyone could help solve the problem.
>
> Thanks for providing the patch.
>
> I used it as the basis of for an update to my reproducible branch where
> I have the other changes waiting for the updated BND release.
>
> I managed to solve the format problem by working with ant.tstamp.now.iso
> rather than ant.tstamp.now. The iso format was relatively simple to
> generate and, as long as you generate a timestamp for UTC, Ant can read it.
I had considered using ant.tstamp.now.iso (you can see I had it in the
patch, actually), but I think it changes the required version of ant for
building. Since this is expected to be used by Tomcat release-managers
only, I think that's a reasonable requirement to put on the ant version.
We may want to put some checks into the build so ensure that if
ant.tstamp.now.iso is set, you have a version of ant which is high
enough to respect it and terminate the build if necessary.
WDYT?
> I'll commit all these changes once BND 6.3.0 is available - hopefully in
> time for the next release round.
Cool.
I haven't played with it in a while, but I seem to recall that if you
execute "pre-release" and "release" at the same time (like "ant
pre-release release"), weird things happen with the timestamp. Also if
you do "pre-release" after having run "pre-release" you get some
weirdness. I think. I'll have to go back and look :)
Finally, "clean" should probably remove the build-release.properties file.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Repeatable builds update
Posted by Mark Thomas <ma...@apache.org>.
On 06/05/2022 14:27, Christopher Schultz wrote:
> Awesome. I have a small commit to build.xml that drops environmental
> information (e.g. versions, etc.) to a build-release.properties file,
> but I'm unable to generate the timestamp in the correct format in order
> to put it into the file. Here's the patch for it; I'd love it if anyone
> could help solve the problem.
Thanks for providing the patch.
I used it as the basis of for an update to my reproducible branch where
I have the other changes waiting for the updated BND release.
I managed to solve the format problem by working with ant.tstamp.now.iso
rather than ant.tstamp.now. The iso format was relatively simple to
generate and, as long as you generate a timestamp for UTC, Ant can read it.
I'll commit all these changes once BND 6.3.0 is available - hopefully in
time for the next release round.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Repeatable builds update
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Mark,
On 5/5/22 15:28, Mark Thomas wrote:
> TL;DR we have platform independent repeatable release builds
Excellent work. Thanks for putting in the time it took for all this.
> And now for the longer version:
>
> Nearly all of the plumbing is in place for this. I have one more commit
> to add to handle repeatable signing of the Windows installer but I am
> holding that back as we need a couple of other things to fall into place
> first.
>
> 1. We need BND 6.3.0. This is because 6.3.0 contains a fix to a
> repeatability issue in the manifest generation. The 6.3.0 release is in
> progress. It is current at RC1 and a final release is expected in a
> couple of weeks.
>
> 2. We need JSign 4.1. I found a couple of minor issues in JSign earlier
> today. I've hacked around them locally to test the repeatable build but
> we need a proper release. ebourg is working on proper fixes to replace
> my hacks. On past experience, I expect he'll have a release out before
> BND does.
>
> Once all of the above is in place our release builds will be repeatable
> on Windows and Linux provided that:
> - The same version of Ant is used
> - The same JDK (vendor and version) is used
>
> The reason for the ANT and JDK version requirements is that the version
> numbers get placed in the manifests.
>
> I have tested this with building on Linux and then repeating the build
> (using the generated signatures for the installer files) on Windows. The
> results are bit for bit identical.
>
> The release process will need a small change. Essentially, the release
> manager will make the usual version changes, call 'ant release' once to
> generate the detached signature files and then tag with the version
> updates and the signature files. Then the release manager calls 'ant
> release' again to create the actual release.
>
> At this point anyone can take the tag and generate the same release
> binaries from the tag.
Awesome. I have a small commit to build.xml that drops environmental
information (e.g. versions, etc.) to a build-release.properties file,
but I'm unable to generate the timestamp in the correct format in order
to put it into the file. Here's the patch for it; I'd love it if anyone
could help solve the problem.
diff --git a/build.xml b/build.xml
index 580b16aca1..88c5f146a1 100644
--- a/build.xml
+++ b/build.xml
@@ -33,6 +33,7 @@
<!-- See "build.properties.default" in the top level directory for
some -->
<!-- property values you may customize.
-->
<property file="${user.home}/build.properties"/>
+ <property file="build-release.properties"/>
<property file="build.properties"/>
<property file="build.properties.default"/>
@@ -2599,6 +2600,44 @@ skip.installer property in build.properties" />
<hashAndSign
file="${tomcat.release}/v${version}/bin/${final.name}.exe" />
</target>
+ <target name="pre-release-read-asf-username"
unless="release.asfusername">
+ <input message="Enter ASF username"
addproperty="release.asfusername" defaultValue="${user.name}" />
+ </target>
+ <target name="pre-release" depends="pre-release-read-asf-username">
+<!--
+ <script language="javascript">
+ project.setProperty('now-sec',
java.lang.System.currentTimeMillis() / 1000);
+ </script>
+-->
+ <tstamp>
+ <format pattern="yyyy-MM-dd'T'HH:mm:ss'Z'"
property="now-readable" timezone="UTC" />
+ </tstamp>
+ <tstamp>
+ <format pattern="ssssssssssssssssss" property="now-sec"
timezone="UTC" />
+ </tstamp>
+ <echo file="build-release.properties">
+# These ant.tstamp properties specify the build-timestamp for
repeatable builds.
+## TODO ant.tstamp.now=${now-sec}
+ant.tstamp.now.iso=${now-readable}
+# This is the ASF user who is the Release Manager for this release.
+release.asfusername=${release.asfusername}
+# Set the version-suffix to "" (empty string) as this is not a
development release.
+version.suffix=
+# Release build environment information:
+release.java.version=${java.version}
+release.java.vm.name=${java.vm.name}
+release.java.vm.version=${java.vm.version}
+release.os=${os.arch} ${os.name} ${os.version}
+release.default.file.encoding=${file.encoding}
+release.ant.version=${ant.version}
+</echo>
+ <echo file="res/maven/mvn.properties">
+asf.ldap.username=${release.asfusername}
+gpg.exec=${gpg.exec}
+maven.asf.release.deploy.version=${version}
+</echo>
+ </target>
+
<target name="release"
depends="clean,release-init,dist-deployer,installer-sign,package-zip,package-winzip,package-tgz,package-deployer-zip,package-deployer-tgz,javadoc,package-docs-tgz,package-src-zip,package-src-tgz,package-src-jar"
description="Create a Tomcat packaged distribution">
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Repeatable builds update
Posted by Emmanuel Bourg <eb...@apache.org>.
Le 05/05/2022 à 21:28, Mark Thomas a écrit :
> TL;DR we have platform independent repeatable release builds
That's really a great achievement, thank you. I hope this will inspire
other Apache projects to make their builds reproducible.
Emmanuel Bourg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org