You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by GitBox <gi...@apache.org> on 2020/07/07 13:59:08 UTC
[GitHub] [commons-lang] diabonas opened a new pull request #578: Improve reproducibility of generated JARs
diabonas opened a new pull request #578:
URL: https://github.com/apache/commons-lang/pull/578
[maven-bundle-plugin](https://felix.apache.org/documentation/subprojects/apache-felix-maven-bundle-plugin-bnd.html) adds a couple of extra manifest headers like `Bnd-LastModified` recording information about the build. This makes it harder to reproduce the JAR bit by bit on a different machine. The documented way to remove these is the `-noextraheaders: true` instruction, see the [bnd-maven-plugin README](https://github.com/bndtools/bnd/blob/cc61fa32da6b779a9607d02b78c2052015493eb6/maven/bnd-maven-plugin/README.md#reproducible-builds) and the [Apache Maven guide to reproducible builds](https://maven.apache.org/guides/mini/guide-reproducible-builds.html).
Even after applying this instruction, the ordering of the `Include-Resource` header appears to be file system dependent. Remove this header as well to allow for reproducible builds.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] garydgregory edited a comment on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
garydgregory edited a comment on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656261143
The better solution would be to offer a PR to the `maven-bundle-plugin` to sort `Include-Resource` entries at the point where these are gathered. I'm not quite sure where in the plugin that happens though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] diabonas commented on a change in pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
diabonas commented on a change in pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#discussion_r452376948
##########
File path: pom.xml
##########
@@ -785,6 +785,12 @@
<version>5.1.1</version>
</dependency>
</dependencies>
+ <configuration>
+ <instructions>
+ <_removeheaders>Bnd-LastModified</_removeheaders>
+ <_removeheaders>Include-Resource</_removeheaders>
Review comment:
Cheers, I fixed the `<_removeheaders>` tag! Regarding the `-noextraheaders: true`, that's what in my first iteration of this PR (b67c7fd53137f43a8051b72df1fa046076c4a5eb), but it [has been suggested to keep the `Tool` header in place](https://github.com/apache/commons-lang/pull/578#issuecomment-656108076), hence the current version only removes `Bnd-LastModified` explicitly. I don't care either way, both solutions would work fine for my purposes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] rotty3000 commented on a change in pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
rotty3000 commented on a change in pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#discussion_r452371269
##########
File path: pom.xml
##########
@@ -785,6 +785,12 @@
<version>5.1.1</version>
</dependency>
</dependencies>
+ <configuration>
+ <instructions>
+ <_removeheaders>Bnd-LastModified</_removeheaders>
+ <_removeheaders>Include-Resource</_removeheaders>
Review comment:
Note that this is the improper syntax. The `-removeheaders` instruction is a comma delimited value. You should use:
```
<_removeheaders>Bnd-LastModified,Include-Resource</_removeheaders>
```
instead.
##########
File path: pom.xml
##########
@@ -785,6 +785,12 @@
<version>5.1.1</version>
</dependency>
</dependencies>
+ <configuration>
+ <instructions>
+ <_removeheaders>Bnd-LastModified</_removeheaders>
+ <_removeheaders>Include-Resource</_removeheaders>
Review comment:
Note that this is improper syntax. The `-removeheaders` instruction is a comma delimited value. You should use:
```
<_removeheaders>Bnd-LastModified,Include-Resource</_removeheaders>
```
instead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] coveralls edited a comment on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-654933994
[](https://coveralls.io/builds/31972508)
Coverage remained the same at 94.598% when pulling **a674cf5f452dbf88c8d055d927196038ca109582 on diabonas:reproducible-jar** into **031f4f61b70ec88d16affc982a7092935fd4a0c9 on apache:master**.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] diabonas commented on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
diabonas commented on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656308648
> You will still need a change here to use version 52-SNAPSHOT of the parent pom (which will have your PR).
Fair point, I'll do this once the upstream PR has been approved and merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] rotty3000 commented on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
rotty3000 commented on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656274739
@bjhargrave true! I can start making a lot of these fixes to freshen mbp's function.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] diabonas closed pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
diabonas closed pull request #578:
URL: https://github.com/apache/commons-lang/pull/578
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] rotty3000 commented on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
rotty3000 commented on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656285701
I have also run the branch of this PR twice locally and the manifests were identical:
```
Manifest-Version: 1.0
Implementation-Title: Apache Commons Lang
Bundle-Description: Apache Commons Lang, a package of Java utility cla
sses for the classes that are in java.lang's hierarchy, or are consi
dered to be so standard as to justify existence in java.lang.
Automatic-Module-Name: org.apache.commons.lang3
Bundle-License: https://www.apache.org/licenses/LICENSE-2.0.txt
Bundle-SymbolicName: org.apache.commons.lang3
Implementation-Version: 3.11-SNAPSHOT
Built-By: rotty
Specification-Vendor: The Apache Software Foundation
Bundle-ManifestVersion: 2
Specification-Title: Apache Commons Lang
Bundle-DocURL: https://commons.apache.org/proper/commons-lang/
Bundle-Vendor: The Apache Software Foundation
Require-Capability: osgi.ee;filter:="(&(osgi.ee=JavaSE)(version=1.8))"
Tool: Bnd-5.1.1.202006162103
Implementation-Vendor: The Apache Software Foundation
Export-Package: org.apache.commons.lang3;version="3.11.0.SNAPSHOT",org
.apache.commons.lang3.arch;version="3.11.0.SNAPSHOT",org.apache.commo
ns.lang3.builder;version="3.11.0.SNAPSHOT",org.apache.commons.lang3.c
ompare;version="3.11.0.SNAPSHOT",org.apache.commons.lang3.concurrent;
version="3.11.0.SNAPSHOT",org.apache.commons.lang3.concurrent.locks;v
ersion="3.11.0.SNAPSHOT",org.apache.commons.lang3.event;version="3.11
.0.SNAPSHOT",org.apache.commons.lang3.exception;version="3.11.0.SNAPS
HOT",org.apache.commons.lang3.function;version="3.11.0.SNAPSHOT",org.
apache.commons.lang3.math;version="3.11.0.SNAPSHOT",org.apache.common
s.lang3.mutable;version="3.11.0.SNAPSHOT",org.apache.commons.lang3.re
flect;version="3.11.0.SNAPSHOT",org.apache.commons.lang3.stream;versi
on="3.11.0.SNAPSHOT",org.apache.commons.lang3.text;version="3.11.0.SN
APSHOT",org.apache.commons.lang3.text.translate;version="3.11.0.SNAPS
HOT",org.apache.commons.lang3.time;version="3.11.0.SNAPSHOT",org.apac
he.commons.lang3.tuple;version="3.11.0.SNAPSHOT"
Bundle-Name: Apache Commons Lang
Bundle-Version: 3.11.0.SNAPSHOT
Build-Jdk-Spec: 1.8
Created-By: Apache Maven Bundle Plugin
Specification-Version: 3.11
Build-Jdk: 1.8.0_252
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] garydgregory commented on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
garydgregory commented on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656721121
Would you all and the maven-bundle-plugin folks mind participating in the thread "Reproducible builds, jars, bundles" on dev@maven.apache.org ? TY.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] diabonas commented on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
diabonas commented on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656272059
> The better solution would be to offer a PR to the `maven-bundle-plugin` to sort `Include-Resource` entries at the point where these are gathered. I'm not quite sure where in the plugin that happens though.
According to the upstream discussion in https://github.com/bndtools/bnd/issues/4221, it seems as if there is no real reason to have the `Include-Resource` at all, in fact it appears to be [some sort of glitch that it is not removed](https://github.com/bndtools/bnd/issues/4221#issuecomment-656246597). If that is the case, I suppose it would be easier to fix the defaults in commons-parent, see https://github.com/bndtools/bnd/issues/4221#issuecomment-656266887 for an (untested) suggestion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] bjhargrave edited a comment on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
bjhargrave edited a comment on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656273624
> finally, if the maven-bundle-plugin itself did:
> `<_removeheaders.defaults>Include-Resource,Private-Package</_removeheaders.defaults>` these would also be added to anything the plugin consumer or parent pom sets.
`maven-bundle-plugin` should not be using `Include-Resource` and `Private-Package` anyway. It should be using `-includeresource` and `-includepackage` (not `-privatepackage` so that `@Export` annotations are honored.).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] garydgregory commented on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
garydgregory commented on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656261143
The better solution would be to offer a PR to the maven-bundle-plugin to sort Include-Resource entries at the point where these are gathered. I'm not quite sure where in the plugin that happens though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] garydgregory commented on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
garydgregory commented on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656264758
Have you tried building 3.11-SNAPSHOT in the same way?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] bjhargrave commented on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
bjhargrave commented on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656273624
> finally, if the maven-bundle-plugin itself did:
> `<_removeheaders.defaults>Include-Resource,Private-Package</_removeheaders.defaults>` these would also be added to anything the plugin consumer or parent pom sets.
'maven-bundle-plugin` should not be using `Include-Resource` and `Private-Package` anyway. It should be using `-includeresource` and `-includepackage` (not `-privatepackage` so that `@Export` annotations are honored.).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] rotty3000 edited a comment on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
rotty3000 edited a comment on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656272161
I think what is happening here is that the maven-bundle-plugin defaults are being overloaded (but in a lossy manner) by the parent pom. the maven bundle-plugin has the defaults for `<_removeheaders>Include-Resource,Private-Package</_removeheaders>` but the parent pom has `<_removeheaders>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME,JAVA_1_9_HOME</_removeheaders>` which loses the plugin defaults.
Both the parent pom and the maven-bundle-plugin are sub-optimal in this regard in that they don't play nicely together, nor do they play nicely with consumers.
The bnd solution to making everyone play nicely are _merged properties_ which allow several participantes to set values that won't simply be overloaded by other applications of the same instruction:
if the parent pom used:
`<_removeheaders.parent>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME,JAVA_1_9_HOME</_removeheaders.parent>`
that would allow the sub-projects free reign over using `<_removeheaders>...</_removeheaders>` without lossy behaviour.
finally, if the maven-bundle-plugin itself did:
`<_removeheaders.defaults>Include-Resource,Private-Package</_removeheaders.defaults>` these would also be added to anything the plugin consumer or parent pom's set.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] diabonas commented on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
diabonas commented on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656283566
I just tested https://github.com/diabonas/commons-parent/commit/8795f4457c3c87a1df09d83d9b2d1462272dac9f and it successfully removes all three headers (`Bnd-LastModified`, `Require-Capability` and `Tool`), making the build reproducible on my machine - so your analysis in https://github.com/apache/commons-lang/pull/578#issuecomment-656272161 that `commons-parent` overrides the `maven-bundle-plugin` defaults appears to be correct :+1:
Given that this appears to be the easiest fix to the problem, I am inclined to just submit the above patch as a PR to `commons-parent`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] garydgregory commented on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
garydgregory commented on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656108076
Hi @diabonas
Thank you for the details.
These sound like bugs in the Maven plugin or more likely the underlying bnd library. I can see that this kind of change will be wanted for all Commons projects. So... we should, IMO, get bnd to fix this or feature it, then make any changes if needed in Commons Parent so all components become reproducible. In the meantime, yeah, we could bring this PR in... but, and please:
- If bnd or the plugin is not following the rules for `Bnd-LastModified` and system properties, please create a bug report there.
- I can't say if the `Include-Resource` issue is a bug or feature request from bnd's POV, but it deserve a ticket there for sure. Either it can be fixed straight up (sort'em) or a feature to optionally sort can be added.
- WRT Tool, it would be nice to exclude this specifically as I thought of this version as part of repro just like using the same Java version and Maven plugins is part of repro.
WDYT?
bnd is here: https://github.com/bndtools/bnd
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] diabonas commented on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
diabonas commented on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656138294
> These sound like bugs in the Maven plugin or more likely the underlying bnd library. I can see that this kind of change will be wanted for all Commons projects. So... we should, IMO, get bnd to fix this or feature it, then make any changes if needed in Commons Parent so all components become reproducible.
That seems to be the better solution indeed, so https://github.com/apache/commons-parent would be the right place to submit this?
> In the meantime, yeah, we could bring this PR in... but, and please:
> * If bnd or the plugin is not following the rules for `Bnd-LastModified` and system properties, please create a bug report there.
I think this part is probably working as intended, from the [upstream Bnd README](https://github.com/bndtools/bnd/blob/cc61fa32da6b779a9607d02b78c2052015493eb6/maven/bnd-maven-plugin/README.md#reproducible-builds) it seems as if having the `Bnd-LastModified` header by default is a deliberate decision, for reproducible builds it is recommended to use `-noextraheaders: true` in order to remove it.
> * I can't say if the `Include-Resource` issue is a bug or feature request from bnd's POV, but it deserve a ticket there for sure. Either it can be fixed straight up (sort'em) or a feature to optionally sort can be added.
Agreed, I opened https://github.com/bndtools/bnd/issues/4221 upstream.
> * WRT Tool, it would be nice to exclude this specifically as I thought of this version as part of repro just like using the same Java version and Maven plugins is part of repro.
The current goal is to be able to reproduce the JAR with the exact same version of the tool chain, so keeping the `Tool` header would be fine by me. This is easily achieved by replacing `<_noextraheaders>true</_noextraheaders>` by `<_removeheaders>Bnd-LastModified</_removeheaders>`, see the updated PR (that should probably directed to Commons Parent instead, see my question above).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] coveralls commented on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
coveralls commented on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-654933994
[](https://coveralls.io/builds/31912464)
Coverage remained the same at 94.679% when pulling **b67c7fd53137f43a8051b72df1fa046076c4a5eb on diabonas:reproducible-jar** into **32a9a350c67f1418784fff266d7e7881ca6346ad on apache:master**.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] diabonas commented on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
diabonas commented on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656294874
OK, I have filed https://github.com/apache/commons-parent/pull/9 as that seems to be the most appropriate place to fix the issue, so I am closing this PR since commons-lang will profit from the upstream PR automatically once commons-parent gets updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] diabonas commented on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
diabonas commented on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656022221
@garydgregory AFAIKT maven-bundle-plugin has no special configuration in version 3.10 at all, so additional manifest headers are included, making it impossible to reproduce a previously built JAR. You can see the difference between two builds of version 3.10 in the following diff, both built with `mvn -Dproject.build.outputTimestamp=1594285478 clean package` to remove other sources of unreproducibility like differing file timestamps:
```
--- commons-lang3-3.10_build1.jar
+++ commons-lang3-3.10_build2.jar
├── zipinfo -v {}
│ @@ -27,15 +27,15 @@
│ minimum file system compatibility required: MS-DOS, OS/2 or NT FAT
│ minimum software version required to extract: 2.0
│ compression method: deflated
│ compression sub-type (deflation): normal
│ file security status: not encrypted
│ extended local header: no
│ file last modified on (DOS date/time): 2020 Jul 9 09:04:38
│ - 32-bit CRC value (hex): 6b84a6c5
│ + 32-bit CRC value (hex): 9bb8ba7e
│ compressed size: 694 bytes
│ uncompressed size: 1869 bytes
│ length of filename: 20 characters
│ length of extra field: 0 bytes
│ length of file comment: 0 characters
│ disk number on which file begins: disk 1
│ apparent file type: binary
├── META-INF/MANIFEST.MF
│ @@ -4,15 +4,15 @@
│ Specification-Title: Apache Commons Lang
│ Specification-Version: 3.10
│ Specification-Vendor: The Apache Software Foundation
│ Implementation-Title: Apache Commons Lang
│ Implementation-Version: 3.10
│ Implementation-Vendor: The Apache Software Foundation
│ Automatic-Module-Name: org.apache.commons.lang3
│ -Bnd-LastModified: 1594285565704
│ +Bnd-LastModified: 1594285992219
│ Build-Jdk: 14.0.1
│ Built-By: builduser
│ Bundle-Description: Apache Commons Lang, a package of Java utility class
│ es for the classes that are in java.lang's hierarchy, or are considere
│ d to be so standard as to justify existence in java.lang.
│ Bundle-DocURL: https://commons.apache.org/proper/commons-lang/
│ Bundle-License: https://www.apache.org/licenses/LICENSE-2.0.txt
│ @@ -27,12 +27,12 @@
│ s.lang3.concurrent;version="3.10",org.apache.commons.lang3.event;versio
│ n="3.10",org.apache.commons.lang3.exception;version="3.10",org.apache.c
│ ommons.lang3.math;version="3.10",org.apache.commons.lang3.mutable;versi
│ on="3.10",org.apache.commons.lang3.reflect;version="3.10",org.apache.co
│ mmons.lang3.text;version="3.10",org.apache.commons.lang3.text.translate
│ ;version="3.10",org.apache.commons.lang3.time;version="3.10",org.apache
│ .commons.lang3.tuple;version="3.10"
│ -Include-Resource: META-INF/LICENSE.txt=LICENSE.txt,META-INF/NOTICE.txt=N
│ - OTICE.txt
│ +Include-Resource: META-INF/NOTICE.txt=NOTICE.txt,META-INF/LICENSE.txt=LI
│ + CENSE.txt
│ Require-Capability: osgi.ee;filter:="(&(osgi.ee=JavaSE)(version=1.8))"
│ Tool: Bnd-4.2.0.201903051501
```
As you can see, `Bnd-LastModified` does not respect [`project.build.outputTimestamp`](https://maven.apache.org/guides/mini/guide-reproducible-builds.html) (or the [`SOURCE_DATE_EPOCH` specification](https://reproducible-builds.org/docs/source-date-epoch/)), making it impossible to get the same JAR each time. Furthermore, `Include-Resource` seems to depend on the file system order (the first build was on a Btrfs file system, the second on a tmpfs file system).
After [backporting](https://git.archlinux.org/svntogit/community.git/tree/trunk/java-commons-lang-3.10_improve-reproducibility-of-generated-JARs.patch?h=packages/java-commons-lang) the patch proposed in this PR to version 3.10, the JAR is fully reproducible between rebuilds since the offending manifest headers are removed. The comparison with the unpatched, unreproducible JAR is as follows:
```
--- commons-lang3-3.10_build1.jar
+++ commons-lang3-3.10_build3-patched.jar
├── zipinfo {}
│ @@ -1,9 +1,9 @@
│ -Zip file size: 526952 bytes, number of entries: 314
│ --rw-r--r-- 2.0 unx 1869 b- defN 20-Jul-09 09:04 META-INF/MANIFEST.MF
│ +Zip file size: 526929 bytes, number of entries: 314
│ +-rw-r--r-- 2.0 unx 1720 b- defN 20-Jul-09 09:04 META-INF/MANIFEST.MF
│ drwxr-xr-x 2.0 unx 0 b- stor 20-Jul-09 09:04 META-INF/
│ drwxr-xr-x 2.0 unx 0 b- stor 20-Jul-09 09:04 org/
│ drwxr-xr-x 2.0 unx 0 b- stor 20-Jul-09 09:04 org/apache/
│ drwxr-xr-x 2.0 unx 0 b- stor 20-Jul-09 09:04 org/apache/commons/
│ drwxr-xr-x 2.0 unx 0 b- stor 20-Jul-09 09:04 org/apache/commons/lang3/
│ drwxr-xr-x 2.0 unx 0 b- stor 20-Jul-09 09:04 org/apache/commons/lang3/arch/
│ drwxr-xr-x 2.0 unx 0 b- stor 20-Jul-09 09:04 org/apache/commons/lang3/builder/
│ @@ -307,10 +307,10 @@
│ -rw-r--r-- 2.0 unx 2443 b- defN 20-Jul-09 09:04 org/apache/commons/lang3/tuple/ImmutableTriple.class
│ -rw-r--r-- 2.0 unx 3065 b- defN 20-Jul-09 09:04 org/apache/commons/lang3/tuple/MutablePair.class
│ -rw-r--r-- 2.0 unx 2636 b- defN 20-Jul-09 09:04 org/apache/commons/lang3/tuple/MutableTriple.class
│ -rw-r--r-- 2.0 unx 1266 b- defN 20-Jul-09 09:04 org/apache/commons/lang3/tuple/Pair$PairAdapter.class
│ -rw-r--r-- 2.0 unx 4228 b- defN 20-Jul-09 09:04 org/apache/commons/lang3/tuple/Pair.class
│ -rw-r--r-- 2.0 unx 1216 b- defN 20-Jul-09 09:04 org/apache/commons/lang3/tuple/Triple$TripleAdapter.class
│ -rw-r--r-- 2.0 unx 3764 b- defN 20-Jul-09 09:04 org/apache/commons/lang3/tuple/Triple.class
│ --rw-r--r-- 2.0 unx 30549 b- defN 20-Jul-09 09:04 META-INF/maven/org.apache.commons/commons-lang3/pom.xml
│ +-rw-r--r-- 2.0 unx 30901 b- defN 20-Jul-09 09:04 META-INF/maven/org.apache.commons/commons-lang3/pom.xml
│ -rw-r--r-- 2.0 unx 65 b- defN 20-Jul-09 09:04 META-INF/maven/org.apache.commons/commons-lang3/pom.properties
│ -314 files, 1150972 bytes uncompressed, 468924 bytes compressed: 59.3%
│ +314 files, 1151175 bytes uncompressed, 468901 bytes compressed: 59.3%
├── META-INF/MANIFEST.MF
│ @@ -4,15 +4,14 @@
│ Specification-Title: Apache Commons Lang
│ Specification-Version: 3.10
│ Specification-Vendor: The Apache Software Foundation
│ Implementation-Title: Apache Commons Lang
│ Implementation-Version: 3.10
│ Implementation-Vendor: The Apache Software Foundation
│ Automatic-Module-Name: org.apache.commons.lang3
│ -Bnd-LastModified: 1594285565704
│ Build-Jdk: 14.0.1
│ Built-By: builduser
│ Bundle-Description: Apache Commons Lang, a package of Java utility class
│ es for the classes that are in java.lang's hierarchy, or are considere
│ d to be so standard as to justify existence in java.lang.
│ Bundle-DocURL: https://commons.apache.org/proper/commons-lang/
│ Bundle-License: https://www.apache.org/licenses/LICENSE-2.0.txt
│ @@ -27,12 +26,9 @@
│ s.lang3.concurrent;version="3.10",org.apache.commons.lang3.event;versio
│ n="3.10",org.apache.commons.lang3.exception;version="3.10",org.apache.c
│ ommons.lang3.math;version="3.10",org.apache.commons.lang3.mutable;versi
│ on="3.10",org.apache.commons.lang3.reflect;version="3.10",org.apache.co
│ mmons.lang3.text;version="3.10",org.apache.commons.lang3.text.translate
│ ;version="3.10",org.apache.commons.lang3.time;version="3.10",org.apache
│ .commons.lang3.tuple;version="3.10"
│ -Include-Resource: META-INF/LICENSE.txt=LICENSE.txt,META-INF/NOTICE.txt=N
│ - OTICE.txt
│ Require-Capability: osgi.ee;filter:="(&(osgi.ee=JavaSE)(version=1.8))"
│ -Tool: Bnd-4.2.0.201903051501
├── META-INF/maven/org.apache.commons/commons-lang3/pom.xml
│ ├── META-INF/maven/org.apache.commons/commons-lang3/pom.xml
│ │ @@ -742,14 +742,24 @@
│ │ <version>4.0.0</version>
│ │ </dependency>
│ │ </dependencies>
│ │ <configuration>
│ │ <excludeFilterFile>${basedir}/spotbugs-exclude-filter.xml</excludeFilterFile>
│ │ </configuration>
│ │ </plugin>
│ │ + <plugin>
│ │ + <groupId>org.apache.felix</groupId>
│ │ + <artifactId>maven-bundle-plugin</artifactId>
│ │ + <configuration>
│ │ + <instructions>
│ │ + <_noextraheaders>true</_noextraheaders>
│ │ + <_removeheaders>Include-Resource</_removeheaders>
│ │ + </instructions>
│ │ + </configuration>
│ │ + </plugin>
│ │ </plugins>
│ │ </build>
│ │ <reporting>
│ │ <plugins>
│ │ <plugin>
│ │ <artifactId>maven-checkstyle-plugin</artifactId>
│ │ <version>${checkstyle.plugin.version}</version>
```
You can see that the headers `Bnd-LastModified`, `Include-Resource` and `Tool` are removed (and that the suggested patch results in a changed `pom.xml`, obviously).
Building version 3.10 with the backported patch results in two completely identical files:
```
$ sha256sum commons-lang3-3.10_build3-patched.jar commons-lang3-3.10_build4-patched.jar
8027ab78b1b8d4698975edf3d855d83e4c14e00e5ee6dcb891290b6c1bfd6d7d commons-lang3-3.10_build3-patched.jar
8027ab78b1b8d4698975edf3d855d83e4c14e00e5ee6dcb891290b6c1bfd6d7d commons-lang3-3.10_build4-patched.jar
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] rotty3000 commented on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
rotty3000 commented on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656272161
I think what is happening here is that the maven-bundle-plugin defaults are being overloaded (but in a lossy manner) by the parent pom. the maven bundle-plugin has the defaults for `<_removeheaders>Include-Resource,Private-Package</_removeheaders>` but the parent pom has `<_removeheaders>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME,JAVA_1_9_HOME</_removeheaders>` which loses the plugin defaults.
Both the parent pom and the maven-bundle-plugin are sub-optimal in this regard in that they don't play nicely together, nor do they play nicely with consumers.
The bnd solution to making everyone play nicely are _merged properties_ which allow several participates to set values that won't by default simply be overloaded by other applications of the same instruction:
if the parent pom used:
`<_removeheaders.parent>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME,JAVA_1_9_HOME</_removeheaders.parent>`
that would allow the sub-projects free reign over using `<_removeheaders>...</_removeheaders>` without lossy behaviour.
finally, if the maven-bundle-plugin itself did:
`<_removeheaders.defaults>Include-Resource,Private-Package</_removeheaders.defaults>` these would also be added to anything the plugin consumer or parent pom's set.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] diabonas commented on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
diabonas commented on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656249174
According to upstream Bnd, either [`maven-bundle-plugin` should switch from `Include-Resource` to `-includeresource`](https://github.com/bndtools/bnd/issues/4221#issuecomment-656158190), or [`removeheaders: Include-Resource` should be used](https://github.com/bndtools/bnd/issues/4221#issuecomment-656159317) to remove the header completely.
I will have a look into the former solution, though I cannot promise anything since I am not familiar with the [code base](https://github.com/apache/felix-dev) at all, the latter solution is what this pull request does.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] garydgregory commented on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
garydgregory commented on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-655859978
Hi @diabonas
What is the difference between what this builds and what we have in 3.10?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] diabonas commented on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
diabonas commented on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656277059
> Have you tried building 3.11-SNAPSHOT in the same way?
have just built the current master 031f4f61b70ec88d16affc982a7092935fd4a0c9 twice using `mvn -Dproject.build.outputTimestamp=1594285478 clean package`, the results are the same as for version 3.10: the headers `Bnd-LastModified`, `Require-Capability` and `Tool` are included in the manifest, the first two making the build unreproducible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] coveralls edited a comment on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-654933994
[](https://coveralls.io/builds/31967771)
Coverage remained the same at 94.598% when pulling **90fc7719940477f68846d009fb531d6e92c63b0f on diabonas:reproducible-jar** into **031f4f61b70ec88d16affc982a7092935fd4a0c9 on apache:master**.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] bjhargrave commented on a change in pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
bjhargrave commented on a change in pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#discussion_r452373785
##########
File path: pom.xml
##########
@@ -785,6 +785,12 @@
<version>5.1.1</version>
</dependency>
</dependencies>
+ <configuration>
+ <instructions>
+ <_removeheaders>Bnd-LastModified</_removeheaders>
+ <_removeheaders>Include-Resource</_removeheaders>
Review comment:
But you are better off using `-noextraheaders: true` to remove `Bnd-LastModified` (and other build sensitive headers) from the manifest and leave the default for `-removedheaders` alone.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] bjhargrave commented on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
bjhargrave commented on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656301973
> so I am closing this PR since commons-lang will profit from the upstream PR automatically once commons-parent gets updated.
You will still need a change here to use version 52-SNAPSHOT of the parent pom (which will have your PR).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] rotty3000 commented on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
rotty3000 commented on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656279602
hey gents if you give me a few moments I have time to check what is going on.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] rotty3000 edited a comment on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
rotty3000 edited a comment on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656272161
I think what is happening here is that the maven-bundle-plugin defaults are being overloaded (but in a lossy manner) by the parent pom. the maven bundle-plugin has the defaults for `<_removeheaders>Include-Resource,Private-Package</_removeheaders>` but the parent pom has `<_removeheaders>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME,JAVA_1_9_HOME</_removeheaders>` which loses the plugin defaults.
Both the parent pom and the maven-bundle-plugin are sub-optimal in this regard in that they don't play nicely together, nor do they play nicely with consumers.
The bnd solution to making everyone play nicely are _merged properties_ which allow several participantes to set values that won't simply be overloaded by other applications of the same instruction:
if the parent pom used:
`<_removeheaders.parent>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME,JAVA_1_9_HOME</_removeheaders.parent>`
that would allow the sub-projects free reign over using `<_removeheaders>...</_removeheaders>` without lossy behaviour.
finally, if the maven-bundle-plugin itself did:
`<_removeheaders.defaults>Include-Resource,Private-Package</_removeheaders.defaults>` these would also be added to anything the plugin consumer or parent pom sets.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] rotty3000 edited a comment on pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
rotty3000 edited a comment on pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#issuecomment-656279602
hey folks if you give me a few moments I have time to check what is going on.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [commons-lang] rotty3000 commented on a change in pull request #578: Improve reproducibility of generated JARs
Posted by GitBox <gi...@apache.org>.
rotty3000 commented on a change in pull request #578:
URL: https://github.com/apache/commons-lang/pull/578#discussion_r452371269
##########
File path: pom.xml
##########
@@ -785,6 +785,12 @@
<version>5.1.1</version>
</dependency>
</dependencies>
+ <configuration>
+ <instructions>
+ <_removeheaders>Bnd-LastModified</_removeheaders>
+ <_removeheaders>Include-Resource</_removeheaders>
Review comment:
Note that this is the improper syntax. The `-remoteheaders` instruction is a comma delimited value. You should use:
```
<_removeheaders>Bnd-LastModified,Include-Resource</_removeheaders>
```
instead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org