You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by GitBox <gi...@apache.org> on 2020/07/09 18:53:10 UTC

[GitHub] [commons-parent] diabonas opened a new pull request #9: Improve reproducibility of generated JARs

diabonas opened a new pull request #9:
URL: https://github.com/apache/commons-parent/pull/9


   [Bnd](https://github.com/bndtools/bnd)/[maven-bundle-plugin](https://felix.apache.org/documentation/subprojects/apache-felix-maven-bundle-plugin-bnd.html) adds a couple of extra manifest headers like `Bnd-LastModified` recording information about the build. This makes it harder to reproduce the JAR bit for bit on a different machine. The [documented way](https://github.com/bndtools/bnd/blob/cc61fa32da6b779a9607d02b78c2052015493eb6/maven/bnd-maven-plugin/README.md#reproducible-builds) to remove these is the `-noextraheaders: true` instruction.
   
   Even after applying this instruction, JARs still include a `Include-Resource` header, whose ordering appears to be file system dependent. It is removed in `maven-bundle-plugin` by default, but the [existing `<_removeheaders>` tag](https://github.com/apache/commons-parent/blob/b39108aba84b93b5b57b8bdc6c934fbadd98521f/pom.xml#L902) overrides
   this. Use merged properties to add headers to be removed instead of replacing the list, as suggested upstream by @rotty3000 in https://github.com/bndtools/bnd/issues/4221#issuecomment-656258021.
   
   This was originally proposed for commons-lang in https://github.com/apache/commons-lang/pull/578, also see the discussion there as well as upstream in https://github.com/bndtools/bnd/issues/4221. Since the problematic `<_removeheaders>` is in commons-parent and other commons-* projects may profit from reproducible builds as well, this seems to be the appropriate place to fix the issue.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [commons-parent] sebbASF commented on a change in pull request #9: Improve reproducibility of generated JARs

Posted by GitBox <gi...@apache.org>.
sebbASF commented on a change in pull request #9:
URL: https://github.com/apache/commons-parent/pull/9#discussion_r452481507



##########
File path: pom.xml
##########
@@ -898,8 +898,10 @@
           <instructions>
             <!-- stops the "uses" clauses being added to "Export-Package" manifest entry -->
             <_nouses>true</_nouses>
+            <!-- remove extra headers to improve bit for bit reproducibility -->
+            <_noextraheaders>true</_noextraheaders>
             <!-- Stop the JAVA_1_n_HOME variables from being treated as headers by Bnd -->
-            <_removeheaders>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME,JAVA_1_9_HOME</_removeheaders>
+            <_removeheaders.commons-parent>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME,JAVA_1_9_HOME</_removeheaders.commons-parent>

Review comment:
       I see.
   
   This is not at all obvious, so it needs to be documented in the pom source with a pointer to the official documentation for the property syntax




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [commons-parent] garydgregory commented on pull request #9: Improve reproducibility of generated JARs

Posted by GitBox <gi...@apache.org>.
garydgregory commented on pull request #9:
URL: https://github.com/apache/commons-parent/pull/9#issuecomment-656659584


   Ah, right, thanks for the clarification.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [commons-parent] garydgregory commented on pull request #9: Improve reproducibility of generated JARs

Posted by GitBox <gi...@apache.org>.
garydgregory commented on pull request #9:
URL: https://github.com/apache/commons-parent/pull/9#issuecomment-656930403


   FYI:
   ```
   ---------- Forwarded message ---------
   From: Romain Manni-Bucau <rm...@gmail.com>
   Date: Fri, Jul 10, 2020 at 10:53 AM
   Subject: Re: Reproducible builds, jars, bundles
   To: Maven Developers List <de...@maven.apache.org>
   
   
   Hi Gary,
   
   Maybe I misunderstood something but isn't it that felix bundle plugin does
   not sort the maven resources ([1]?
   
   [1]
   https://github.com/apache/felix-dev/blob/7f4d31b384d9d83c772680a8627df18ff078eaa4/tools/maven-bundle-plugin/src/main/java/org/apache/felix/bundleplugin/BundlePlugin.java#L2036
   
   Romain Manni-Bucau
   @rmannibucau <https://twitter.com/rmannibucau> |  Blog
   <https://rmannibucau.metawerx.net/> | Old Blog
   <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
   LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
   <https://www.packtpub.com/application-development/java-ee-8-high-performance>
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [commons-parent] rotty3000 commented on a change in pull request #9: Improve reproducibility of generated JARs

Posted by GitBox <gi...@apache.org>.
rotty3000 commented on a change in pull request #9:
URL: https://github.com/apache/commons-parent/pull/9#discussion_r452581524



##########
File path: pom.xml
##########
@@ -898,8 +898,10 @@
           <instructions>
             <!-- stops the "uses" clauses being added to "Export-Package" manifest entry -->
             <_nouses>true</_nouses>
+            <!-- remove extra headers to improve bit for bit reproducibility -->
+            <_noextraheaders>true</_noextraheaders>
             <!-- Stop the JAVA_1_n_HOME variables from being treated as headers by Bnd -->
-            <_removeheaders>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME,JAVA_1_9_HOME</_removeheaders>
+            <_removeheaders.commons-parent>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME,JAVA_1_9_HOME</_removeheaders.commons-parent>

Review comment:
       The suffix is a user defined descriminator. The value is entirely informational but should be something unique-enough, that conveys some meaning for the use case. In this case it's something like _these are the `-removeheaders` properties set by the `commons-parent` pom_.
   
   If you think the docs page can better describe this I can take a shot at making it more clear.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [commons-parent] sebbASF commented on a change in pull request #9: Improve reproducibility of generated JARs

Posted by GitBox <gi...@apache.org>.
sebbASF commented on a change in pull request #9:
URL: https://github.com/apache/commons-parent/pull/9#discussion_r452458794



##########
File path: pom.xml
##########
@@ -898,8 +898,10 @@
           <instructions>
             <!-- stops the "uses" clauses being added to "Export-Package" manifest entry -->
             <_nouses>true</_nouses>
+            <!-- remove extra headers to improve bit for bit reproducibility -->
+            <_noextraheaders>true</_noextraheaders>
             <!-- Stop the JAVA_1_n_HOME variables from being treated as headers by Bnd -->
-            <_removeheaders>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME,JAVA_1_9_HOME</_removeheaders>
+            <_removeheaders.commons-parent>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME,JAVA_1_9_HOME</_removeheaders.commons-parent>

Review comment:
       I don't understand what _removeheaders.commons-parent is doing?
   Why is it necessary to change that line?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [commons-parent] diabonas commented on a change in pull request #9: Improve reproducibility of generated JARs

Posted by GitBox <gi...@apache.org>.
diabonas commented on a change in pull request #9:
URL: https://github.com/apache/commons-parent/pull/9#discussion_r452497087



##########
File path: pom.xml
##########
@@ -898,8 +898,10 @@
           <instructions>
             <!-- stops the "uses" clauses being added to "Export-Package" manifest entry -->
             <_nouses>true</_nouses>
+            <!-- remove extra headers to improve bit for bit reproducibility -->
+            <_noextraheaders>true</_noextraheaders>
             <!-- Stop the JAVA_1_n_HOME variables from being treated as headers by Bnd -->
-            <_removeheaders>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME,JAVA_1_9_HOME</_removeheaders>
+            <_removeheaders.commons-parent>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME,JAVA_1_9_HOME</_removeheaders.commons-parent>

Review comment:
       Done, the revised version of the PR has a description that should hopefully be more clear.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [commons-parent] sebbASF commented on a change in pull request #9: Improve reproducibility of generated JARs

Posted by GitBox <gi...@apache.org>.
sebbASF commented on a change in pull request #9:
URL: https://github.com/apache/commons-parent/pull/9#discussion_r452761612



##########
File path: pom.xml
##########
@@ -898,8 +898,10 @@
           <instructions>
             <!-- stops the "uses" clauses being added to "Export-Package" manifest entry -->
             <_nouses>true</_nouses>
+            <!-- remove extra headers to improve bit for bit reproducibility -->
+            <_noextraheaders>true</_noextraheaders>
             <!-- Stop the JAVA_1_n_HOME variables from being treated as headers by Bnd -->
-            <_removeheaders>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME,JAVA_1_9_HOME</_removeheaders>
+            <_removeheaders.commons-parent>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME,JAVA_1_9_HOME</_removeheaders.commons-parent>

Review comment:
       Thanks.
   The docs are not clear on how Bind merges properties.
   For example, if a child pom uses the same discriminator as a parent pom, are the values merged or replaced, and if replaced, which takes precedence? This will obviously affect the choice of suffix.
   It would be useful if the documentation provided some examples.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [commons-parent] sebbASF commented on a change in pull request #9: Improve reproducibility of generated JARs

Posted by GitBox <gi...@apache.org>.
sebbASF commented on a change in pull request #9:
URL: https://github.com/apache/commons-parent/pull/9#discussion_r452529894



##########
File path: pom.xml
##########
@@ -898,8 +898,10 @@
           <instructions>
             <!-- stops the "uses" clauses being added to "Export-Package" manifest entry -->
             <_nouses>true</_nouses>
+            <!-- remove extra headers to improve bit for bit reproducibility -->
+            <_noextraheaders>true</_noextraheaders>
             <!-- Stop the JAVA_1_n_HOME variables from being treated as headers by Bnd -->
-            <_removeheaders>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME,JAVA_1_9_HOME</_removeheaders>
+            <_removeheaders.commons-parent>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME,JAVA_1_9_HOME</_removeheaders.commons-parent>

Review comment:
       Thanks, I've read the page in question, and it mentions merged properties but does not describe them as far as I can tell.
   
   In particular it does not describe what the .commons-parent suffix represents.
   It looks as though it may be related to the containing pom.
   Is it the artifactId?
   If so, why is ${project.artifactId} not used?
   If not, how is the value chosen?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [commons-parent] diabonas commented on a change in pull request #9: Improve reproducibility of generated JARs

Posted by GitBox <gi...@apache.org>.
diabonas commented on a change in pull request #9:
URL: https://github.com/apache/commons-parent/pull/9#discussion_r452463447



##########
File path: pom.xml
##########
@@ -898,8 +898,10 @@
           <instructions>
             <!-- stops the "uses" clauses being added to "Export-Package" manifest entry -->
             <_nouses>true</_nouses>
+            <!-- remove extra headers to improve bit for bit reproducibility -->
+            <_noextraheaders>true</_noextraheaders>
             <!-- Stop the JAVA_1_n_HOME variables from being treated as headers by Bnd -->
-            <_removeheaders>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME,JAVA_1_9_HOME</_removeheaders>
+            <_removeheaders.commons-parent>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME,JAVA_1_9_HOME</_removeheaders.commons-parent>

Review comment:
       See the explanations in https://github.com/bndtools/bnd/issues/4221#issuecomment-656258021 and https://github.com/apache/commons-lang/pull/578#issuecomment-656272161: `<_removeheaders>` overrides the defaults from `maven-bundle-plugin`, thus removing some values from this list that should have stayed there, in particular `Include-Resource`. On the other hand, using the *merged property* `<_removeheaders.commons-parent>` appends to the defaults, adding the values listed here to the ones already present in `maven-bundle-plugin`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [commons-parent] garydgregory commented on pull request #9: Improve reproducibility of generated JARs

Posted by GitBox <gi...@apache.org>.
garydgregory commented on pull request #9:
URL: https://github.com/apache/commons-parent/pull/9#issuecomment-656652055


   Continuing what we had talked about in https://github.com/apache/commons-lang/pull/578 ...
   In the longer term, is there not a fix we can put in the `maven-bundle-plugin` to create the `Include-Resource` in sorted order?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [commons-parent] diabonas commented on pull request #9: Improve reproducibility of generated JARs

Posted by GitBox <gi...@apache.org>.
diabonas commented on pull request #9:
URL: https://github.com/apache/commons-parent/pull/9#issuecomment-656657751


   > In the longer term, is there not a fix we can put in the `maven-bundle-plugin` to create the `Include-Resource` in sorted order?
   
   According to https://github.com/bndtools/bnd/issues/4221#issuecomment-656284086, the ordering issue might not even be in `maven-bundle-plugin`, but in Maven itself. I have had a look over the `maven-bundle-plugin` code as well, but couldn't find a place where a file system dependent reordering might take place either.
   
   Anyway, I think the fix proposed here is correct in itself, `commons-parent` shouldn't overwrite the `maven-bundle-plugin` defaults even if that didn't pose a reproducibility issue.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [commons-parent] garydgregory commented on a change in pull request #9: Improve reproducibility of generated JARs

Posted by GitBox <gi...@apache.org>.
garydgregory commented on a change in pull request #9:
URL: https://github.com/apache/commons-parent/pull/9#discussion_r452898567



##########
File path: pom.xml
##########
@@ -898,8 +898,10 @@
           <instructions>
             <!-- stops the "uses" clauses being added to "Export-Package" manifest entry -->
             <_nouses>true</_nouses>
+            <!-- remove extra headers to improve bit for bit reproducibility -->
+            <_noextraheaders>true</_noextraheaders>
             <!-- Stop the JAVA_1_n_HOME variables from being treated as headers by Bnd -->
-            <_removeheaders>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME,JAVA_1_9_HOME</_removeheaders>
+            <_removeheaders.commons-parent>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME,JAVA_1_9_HOME</_removeheaders.commons-parent>

Review comment:
       Would you all and the maven-bundle-plugin folks mind participating in the thread "Reproducible builds, jars, bundles" on dev@maven.apache.org ? TY.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org