You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Hervé BOUTEMY <he...@free.fr> on 2019/11/01 08:23:36 UTC

Re: last review of Reproducible Builds proposal

Le jeudi 31 octobre 2019, 17:26:52 CET Christofer Dutz a écrit :
> Hi all,
> 
> as I can see you're voting on releasing the reproducible build extended
> plugin versions.
 Is there any documentation on how to use this new
> feature?
> 
> I had a look at the confluence page, but that seemed like a brainstorming
> session.
ok, the Wiki page [1] started as a brainstorming session, was updated to a proposal (the "Output Archive Entries Timestamp" parapgraph).
And now I probably should order paragraph a little bit, and add a "Making your build reproducible" section for end uses to have a quick explanation.

I'll write the explanation here as a first try before working on the Wiki:

1. upgrade your plugins to reproducible version, particularly mpaven-source-plugin, maven-jar-plugin and maven-assembly-plugin to vesion 3.2.0
2. add project.build.outputTimestamp property with the timestamp value that will be used in zip/jar/tar archives:
  <properties>
    <project.build.outputTimestamp>2019-10-02T08:04:00Z</project.build.outputTimestamp>
  </properties>

Notice:
- there is no Maven version prerequisite, everything happens at plugins level
- Reproducible Builds require to have no version ranges in dependencies, generally gives different result on Unixes vs Windows, and generally depend on the major version of JDK used to compile (see "Out of Scope" paragraph)

You have the basis configured, the output should be reproducible now.
If something is still not reproducible, use diffoscope to find the unstable output, find the plugin that generated this output and check if there is a reproducible version available: if not, please open an issue to help plugin maintainers improving Reproducible Builds support at every plugin level.

[1]  https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318


 
> I would love to add this to the PLC4X build asap.
I'd love to have feedback
Don't hesitate to ping me.
> 
> So I would like to test the release-candidates and vote too.
I would love to have many tester and votes :)

> 
> Chris
> 
> 
> 
> Am 16.10.19, 14:42 schrieb "Hervé BOUTEMY" <he...@free.fr>:
> 
>     Le mercredi 16 octobre 2019, 13:40:48 CEST Andreas Sewe a écrit :
> 
>     > Emmanuel Bourg wrote:
>     > 
>     > > Le 16/10/2019 à 08:35, Hervé BOUTEMY a écrit :
>     > > 
>     > >> last question: now that we seem to fully understand each other,
>     > >> does it
>     > >> mean that you don't need any more "seconds since the epoch" format
>     > >> support for the property?
>     > > 
>     > > 
>     > > If Maven supports the SOURCE_DATE_EPOCH environment variable I
>     > > don't
>     > > think that's necessary, otherwise it would be nice to be able to
>     > > invoke
>     > > 
>     > > Maven with:
>     > > 
>     > >    mvn package -Dproject.build.outputTimestamp=$SOURCE_DATE_EPOCH
>     > > 
>     > > 
>     > > and this means supporting a timestamp formatted as seconds since
>     > > the
>     > > epoch.
>     > 
>     > 
>     > +1
>     > 
>     > The above would be a nice, simple way of bridging the gap between
>     > SOURCE_DATE_EPOCH and project.build.outputTimestamp.
> 
>     told like that, ok, why not
>     
> 
>     > 
>     > If it is not too much trouble to implement the "\d+ -> seconds since
>     > epoch" heuristic, them I would love to see it included.
> 
>     ok, I'll do and prepare the release
>     
>     Regards,
>     
>     Hervé
>     
> 
>     > 
>     > Best wishes,
>     > 
>     > Andreas
> 
>     
>     
>     
>     
>     
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>     For additional commands, e-mail: dev-help@maven.apache.org
>     
>     
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
> 





---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Re: last review of Reproducible Builds proposal

Posted by Andreas Sewe <se...@st.informatik.tu-darmstadt.de>.
Hi,

> but the format of the timestamp is completely different ... doesn't that matter?

I don't think so; at least I don't think the Bnd-LastModified header is
meant to be parsed by machines.

But I have _removeheaders>Bnd-*</_removeheaders> (yes, wildcards are
supported) in my <instructions>, so what do I know...

> I was currently going for the option number 1 with removing the header.
> 
> In order to be 100% happy with this, I would prefer a setup where the normal mechanisms are used if no maven.build.outputTimestap is defined, but if it is (because a future version of the maven release plugin set it there) it uses that.

You could use a <profile> that is activated when
maven.build.outputTimestap is defined (or not defined) to implement this
behavior. I haven't tested this, however.

Best wishes,

Andreas




AW: last review of Reproducible Builds proposal

Posted by Christofer Dutz <ch...@c-ware.de>.
Hi all,

And I might want to add, that includes pom, jar, sources jar, javadoc, release-asembly and fat-jars ...

Currently applying these changes to other apache projects as an experiment.

So is anyone already working on the release plugin?

Chris
________________________________
Von: Christofer Dutz <ch...@c-ware.de>
Gesendet: Montag, 4. November 2019 18:58:10
An: Maven Developers List <de...@maven.apache.org>
Betreff: Re: last review of Reproducible Builds proposal

Hi all,

so today I made sure the LastModified and the Creator username are omitted and now the Apache PLC4X build had a 100% reproducible build (at least on my one machine) ... will be checking this out on other machines.

Chris

Am 04.11.19, 10:12 schrieb "Christofer Dutz" <ch...@c-ware.de>:

    Hi Andreas,

    but the format of the timestamp is completely different ... doesn't that matter?
    I was currently going for the option number 1 with removing the header.

    In order to be 100% happy with this, I would prefer a setup where the normal mechanisms are used if no maven.build.outputTimestap is defined, but if it is (because a future version of the maven release plugin set it there) it uses that.

    Will try out your suggestions as soon as I'm able to build again (I unfortunately installed that Mac OS update ... now things aren't working as they should be)

    Chris



    Am 04.11.19, 09:42 schrieb "Andreas Sewe" <se...@st.informatik.tu-darmstadt.de>:

        Christofer Dutz wrote:
        > Well I have a new candidate:
        >
        > <artifactId>maven-bundle-plugin</artifactId>
        >
        > Creates: Bnd-LastModified in the MANIFEST.MF
        >
        > Gotta find out a way to either suppress that entry (Would be great if it could consume the same property)

        <configuration>
          <instructions>
            <_removeheaders>Bnd-LastModified</_removeheaders>
          </instructions>
        <configuration>

        Alternatively

        <configuration>
          <instructions>
            <Bnd-LastModified>${maven.build.outputTimestap}</Bnd-LastModified>
          </instructions>
        <configuration

        Hope that helps,

        Andreas

        ---------------------------------------------------------------------
        To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
        For additional commands, e-mail: dev-help@maven.apache.org




    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
    For additional commands, e-mail: dev-help@maven.apache.org




Re: last review of Reproducible Builds proposal

Posted by Christofer Dutz <ch...@c-ware.de>.
Hi all,

so today I made sure the LastModified and the Creator username are omitted and now the Apache PLC4X build had a 100% reproducible build (at least on my one machine) ... will be checking this out on other machines.

Chris

Am 04.11.19, 10:12 schrieb "Christofer Dutz" <ch...@c-ware.de>:

    Hi Andreas,
    
    but the format of the timestamp is completely different ... doesn't that matter?
    I was currently going for the option number 1 with removing the header.
    
    In order to be 100% happy with this, I would prefer a setup where the normal mechanisms are used if no maven.build.outputTimestap is defined, but if it is (because a future version of the maven release plugin set it there) it uses that.
    
    Will try out your suggestions as soon as I'm able to build again (I unfortunately installed that Mac OS update ... now things aren't working as they should be)
    
    Chris
    
    
    
    Am 04.11.19, 09:42 schrieb "Andreas Sewe" <se...@st.informatik.tu-darmstadt.de>:
    
        Christofer Dutz wrote:
        > Well I have a new candidate:
        > 
        > <artifactId>maven-bundle-plugin</artifactId>
        >  
        > Creates: Bnd-LastModified in the MANIFEST.MF
        > 
        > Gotta find out a way to either suppress that entry (Would be great if it could consume the same property)
        
        <configuration>
          <instructions>
            <_removeheaders>Bnd-LastModified</_removeheaders>
          </instructions>
        <configuration>
        
        Alternatively
        
        <configuration>
          <instructions>
            <Bnd-LastModified>${maven.build.outputTimestap}</Bnd-LastModified>
          </instructions>
        <configuration
        
        Hope that helps,
        
        Andreas
        
        ---------------------------------------------------------------------
        To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
        For additional commands, e-mail: dev-help@maven.apache.org
        
        
    
    
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
    For additional commands, e-mail: dev-help@maven.apache.org
    
    


Re: last review of Reproducible Builds proposal

Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Andreas,

but the format of the timestamp is completely different ... doesn't that matter?
I was currently going for the option number 1 with removing the header.

In order to be 100% happy with this, I would prefer a setup where the normal mechanisms are used if no maven.build.outputTimestap is defined, but if it is (because a future version of the maven release plugin set it there) it uses that.

Will try out your suggestions as soon as I'm able to build again (I unfortunately installed that Mac OS update ... now things aren't working as they should be)

Chris



Am 04.11.19, 09:42 schrieb "Andreas Sewe" <se...@st.informatik.tu-darmstadt.de>:

    Christofer Dutz wrote:
    > Well I have a new candidate:
    > 
    > <artifactId>maven-bundle-plugin</artifactId>
    >  
    > Creates: Bnd-LastModified in the MANIFEST.MF
    > 
    > Gotta find out a way to either suppress that entry (Would be great if it could consume the same property)
    
    <configuration>
      <instructions>
        <_removeheaders>Bnd-LastModified</_removeheaders>
      </instructions>
    <configuration>
    
    Alternatively
    
    <configuration>
      <instructions>
        <Bnd-LastModified>${maven.build.outputTimestap}</Bnd-LastModified>
      </instructions>
    <configuration
    
    Hope that helps,
    
    Andreas
    
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
    For additional commands, e-mail: dev-help@maven.apache.org
    
    


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Re: last review of Reproducible Builds proposal

Posted by Andreas Sewe <se...@st.informatik.tu-darmstadt.de>.
Christofer Dutz wrote:
> Well I have a new candidate:
> 
> <artifactId>maven-bundle-plugin</artifactId>
>  
> Creates: Bnd-LastModified in the MANIFEST.MF
> 
> Gotta find out a way to either suppress that entry (Would be great if it could consume the same property)

<configuration>
  <instructions>
    <_removeheaders>Bnd-LastModified</_removeheaders>
  </instructions>
<configuration>

Alternatively

<configuration>
  <instructions>
    <Bnd-LastModified>${maven.build.outputTimestap}</Bnd-LastModified>
  </instructions>
<configuration

Hope that helps,

Andreas

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Re: last review of Reproducible Builds proposal

Posted by Christofer Dutz <ch...@c-ware.de>.
Well I have a new candidate:

<artifactId>maven-bundle-plugin</artifactId>
 
Creates: Bnd-LastModified in the MANIFEST.MF

Gotta find out a way to either suppress that entry (Would be great if it could consume the same property)

Chris



Am 03.11.19, 20:25 schrieb "herve.boutemy@free.fr" <he...@free.fr>:

    ok, Reproducible Builds are not so easy to get: each plugin that you use can cause an issue
    
    I really recommend you diffoscope to investigate differences, it really helps a lot by easily giving you precise differences
    
    Good luck for the investigations :)
    
    Regards,
    
    Hervé
    
    ----- Mail original -----
    De: "Christofer Dutz" <ch...@c-ware.de>
    À: "Maven Developers List" <de...@maven.apache.org>
    Envoyé: Dimanche 3 Novembre 2019 20:16:42
    Objet: Re: last review of Reproducible Builds proposal
    
    Hi Herve,
    
    unfortunately I now have implemented some tooling to compare the stuff produced by the updated reproducible builds.
    And it does show quite a number of non binary equal files.
    
    Will investigate what's the difference.
    
    Chris
    
    
    
    Am 03.11.19, 11:08 schrieb "Hervé BOUTEMY" <he...@free.fr>:
    
        great feedback, thank you: this proves the initial set of only 3 plugins is a 
        good basis.
        For checking the many output artifacts from a build, future buildinfo file [1] 
        should help a lot. I still have many usability concerns for Maven multi-module 
        builds (starting with: should we generate only the root buildinfo or one 
        buildinfo per Maven module?)
        Let's keep the ball rolling: today, it's plugins release day!!!
        
        Regards,
        
        Hervé
        
        [1] https://reproducible-builds.org/docs/jvm/
        
        Le vendredi 1 novembre 2019, 13:34:32 CET Christofer Dutz a écrit :
        > Hi,
        > 
        > so I just updated the versions of the 3 plugins and gave the Apache PLC4X
        > build a little spin ...
        > https://github.com/apache/plc4x/tree/feature/reproducible-builds
        > 
        > Did two executions of this: 
        > mvn -U -P
        > with-boost,with-java,with-dotnet,with-cpp,with-python,with-proxies,with-san
        > dbox,with
        > -DaltDeploymentRepository=snapshot-repo::default::file:./local-snapshots-di
        > r clean deploy
         (but with a different second local repo location)
        > 
        > Then I used "diff" and "cmp" to compare individual files and it didn't show
        > differences with the artifacts I chose ... 
         now I guess I'll have to whip
        > up some little bash script to sort of compare the artifacts from one
        > directory with the other (Unfortunately the SNAPSHOT timestamps are a
        > little annoying :-/ 
        > We do have some C++, C# and Python modules ... but I wouldn't expect the C++
        > and C# to be reproducible.
         
        > Chris
        > 
        > 
        > Am 01.11.19, 12:04 schrieb "Hervé BOUTEMY" <he...@free.fr>:
        > 
        >     Le vendredi 1 novembre 2019, 09:40:40 CET Christofer Dutz a écrit :
        > 
        >     > Hi Herve,
        >     > 
        >     > thanks for that will try it asap and report any findings back.
        >     > 
        >     > But good to know that there is a difference between JDK major versions
        >     > and
        >     > OSes ... so it would probably be best to stage releases on Linux with
        >     > an
        >     > OpenJDK of the minimum supported version?
        >     > Just thinking how to make it
        >     > possible to verify without having to buy Mac or Windows licenses ...
        >     > guess
        >     > on every machine you could whip up a Ubuntu VM for verification. Just
        >     > thinking about it ... perhaps it would be best to create a Docker
        >     > image for
         doing the reproducible stuff ...
        > 
        >     just to be clear: the difference is on newline only, then Windows vs
        > anything 
         else. You get the same result on Linux, MacOS, FreeBSD, or any
        > other Unix. 
        >     And if I want to be complete, if you get a source tarball from Windows,
        > 
         extract it to a Linux/MacOS/... box and build with "mvn -
        >     Dline.separator='\r\n'", in general, you get the same result as a build
        > under 
         Windows: I tested multiple times, it worked, but we'll see if it
        > works always or just "in general"
        >     
        > 
        >     > Are there any plans on creating a plugin to allow verification?
        >     > 
        >     > Sort of something like this:
        >     > "mvn package release:verify-reproduicble
        >     > -DstagingRepoUrl=a.b.c.de/repo/blahblahblah"
        >     > (Which doesn't deploy the
        >     > artifacts, but instead download them and do a binary comparison) 
        > 
        >     yes, but for now it was completely another form: this is the "Buildinfo
        > file" 
         proposal https://reproducible-builds.org/docs/jvm/
        >     that I stopped to work on 1 year ago given I had no chance to get the
        > same 
         output: it's now a good time to restart working on it as next steps 
        >     
        > 
        >     > Also it could be great if the release-plugin could automatically set
        >     > the
        >     > property:
        >     > a) if it finds the "project.build.outputTimestamp" set to some
        >     > placeholder value b) if some switch tells it to prepare a
        >     > reproducible
        >     > build by using some sort of "switch" parameter 
        >     > Guess that would sort of close the loop to get the biggest benefit out
        >     > of
        >     > the reproducible builds.
        > 
        >     +1
        >     issue has been created
        > https://issues.apache.org/jira/browse/MRELEASE-1029
         But I didn't work on
        > it yet, help welcome
        >     
        > 
        >     > I would be happy to help as I think this is one
        >     > of the greatest new features. (Ok ... perhaps besides the
        >     > sound-output-extension I learned about yesterday ;-) ) 
        > 
        >     +1
        >     the current step is important, but it's only the beginning of the story
        > from 
         an effective usage perspective
        >     
        >     Regards,
        >     
        >     Hervé
        >     
        > 
        >     > 
        >     > Chris
        >     > 
        >     > 
        >     > Am 01.11.19, 09:24 schrieb "Hervé BOUTEMY" <he...@free.fr>:
        >     > 
        >     > 
        >     >     Le jeudi 31 octobre 2019, 17:26:52 CET Christofer Dutz a écrit :
        >     > 
        >     > 
        >     > 
        >     >     > Hi all,
        >     >     > 
        >     >     > as I can see you're voting on releasing the reproducible build
        >     >     > extended
        >     >     > plugin versions.
        >     > 
        >     > 
        >     > 
        >     >      Is there any documentation on how to use this new
        >     > 
        >     > 
        >     > 
        >     >     > feature?
        >     >     > 
        >     >     > I had a look at the confluence page, but that seemed like a
        >     >     > brainstorming
        >     >     > session.
        >     > 
        >     > 
        >     > 
        >     >     ok, the Wiki page [1] started as a brainstorming session, was
        >     >     updated to
        >     > 
        >     > a proposal (the "Output Archive Entries Timestamp" parapgraph).
        > 
        >      And now I
        > 
        >     > probably should order paragraph a little bit, and add a "Making your
        >     > build
        >     > reproducible" section for end uses to have a quick explanation. 
        >     > 
        >     >     I'll write the explanation here as a first try before working on
        >     >     the
        >     > 
        >     > Wiki:
        > 
        >      
        > 
        >     >     1. upgrade your plugins to reproducible version, particularly
        >     > 
        >     > mpaven-source-plugin, maven-jar-plugin and maven-assembly-plugin to
        >     > vesion
        >     > 3.2.0
        > 
        >      2. add project.build.outputTimestamp property with the timestamp
        > 
        >     > value that will be used in zip/jar/tar archives: <properties>
        >     > 
        >     >        
        >     > 
        >     > <project.build.outputTimestamp>2019-10-02T08:04:00Z</project.build.out
        >     > putTi
         mestamp>
        > 
        >      </properties>
        > 
        >     >     
        >     >     Notice:
        >     >     - there is no Maven version prerequisite, everything happens at
        >     >     plugins
        >     > 
        >     > level
        > 
        >      - Reproducible Builds require to have no version ranges in
        > 
        >     > dependencies, generally gives different result on Unixes vs Windows,
        >     > and
        >     > generally depend on the major version of JDK used to compile (see "Out
        >     > of
        >     > Scope" paragraph) 
        >     > 
        >     >     You have the basis configured, the output should be reproducible
        >     >     now.
        >     >     If something is still not reproducible, use diffoscope to find
        >     >     the
        >     > 
        >     > unstable output, find the plugin that generated this output and check
        >     > if
        >     > there is a reproducible version available: if not, please open an
        >     > issue to
        >     > help plugin maintainers improving Reproducible Builds support at
        >     > every
        >     > plugin level.
        > 
        >      
        > 
        >     >     [1] 
        >     > 
        >     > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682
        >     > 318
        > 
        >      
        > 
        >     >     
        >     >     
        >     >      
        >     > 
        >     > 
        >     > 
        >     >     > I would love to add this to the PLC4X build asap.
        >     > 
        >     > 
        >     > 
        >     >     I'd love to have feedback
        >     >     Don't hesitate to ping me.
        >     > 
        >     > 
        >     > 
        >     >     > 
        >     >     > So I would like to test the release-candidates and vote too.
        >     > 
        >     > 
        >     > 
        >     >     I would love to have many tester and votes :)
        >     >     
        >     > 
        >     > 
        >     > 
        >     >     > 
        >     >     > Chris
        >     >     > 
        >     >     > 
        >     >     > 
        >     >     > Am 16.10.19, 14:42 schrieb "Hervé BOUTEMY"
        >     >     > <he...@free.fr>:
        >     >     > 
        >     >     > 
        >     >     > 
        >     >     >     Le mercredi 16 octobre 2019, 13:40:48 CEST Andreas Sewe a
        >     >     >     écrit :
        >     >     > 
        >     >     > 
        >     >     > 
        >     >     > 
        >     >     > 
        >     >     >     > Emmanuel Bourg wrote:
        >     >     >     > 
        >     >     >     > 
        >     >     >     > 
        >     >     >     > > Le 16/10/2019 à 08:35, Hervé BOUTEMY a écrit :
        >     >     >     > > 
        >     >     >     > > 
        >     >     >     > > 
        >     >     >     > >> last question: now that we seem to fully understand
        >     >     >     > >> each
        >     >     >     > >> other,
        >     >     >     > >> does it
        >     >     >     > >> mean that you don't need any more "seconds since the
        >     >     >     > >> epoch"
        >     >     >     > >> format
        >     >     >     > >> support for the property?
        >     >     >     > > 
        >     >     >     > > 
        >     >     >     > > 
        >     >     >     > > 
        >     >     >     > > If Maven supports the SOURCE_DATE_EPOCH environment
        >     >     >     > > variable
        >     >     >     > > I
        >     >     >     > > don't
        >     >     >     > > think that's necessary, otherwise it would be nice to be
        >     >     >     > > able
        >     >     >     > > to
        >     >     >     > > invoke
        >     >     >     > > 
        >     >     >     > > Maven with:
        >     >     >     > > 
        >     >     >     > > 
        >     >     >     > > 
        >     >     >     > >    mvn package
        >     >     >     > >    -Dproject.build.outputTimestamp=$SOURCE_DATE_EPOCH
        >     >     >     > > 
        >     >     >     > > 
        >     >     >     > > 
        >     >     >     > > 
        >     >     >     > > and this means supporting a timestamp formatted as
        >     >     >     > > seconds
        >     >     >     > > since
        >     >     >     > > the
        >     >     >     > > epoch.
        >     >     >     > 
        >     >     >     > 
        >     >     >     > 
        >     >     >     > 
        >     >     >     > +1
        >     >     >     > 
        >     >     >     > The above would be a nice, simple way of bridging the gap
        >     >     >     > between
        >     >     >     > SOURCE_DATE_EPOCH and project.build.outputTimestamp.
        >     >     > 
        >     >     > 
        >     >     > 
        >     >     > 
        >     >     > 
        >     >     >     told like that, ok, why not
        >     >     >     
        >     >     > 
        >     >     > 
        >     >     > 
        >     >     > 
        >     >     > 
        >     >     >     > 
        >     >     >     > If it is not too much trouble to implement the "\d+ ->
        >     >     >     > seconds
        >     >     >     > since
        >     >     >     > epoch" heuristic, them I would love to see it included.
        >     >     > 
        >     >     > 
        >     >     > 
        >     >     > 
        >     >     > 
        >     >     >     ok, I'll do and prepare the release
        >     >     >     
        >     >     >     Regards,
        >     >     >     
        >     >     >     Hervé
        >     >     >     
        >     >     > 
        >     >     > 
        >     >     > 
        >     >     > 
        >     >     > 
        >     >     >     > 
        >     >     >     > Best wishes,
        >     >     >     > 
        >     >     >     > Andreas
        >     >     > 
        >     >     > 
        >     >     > 
        >     >     > 
        >     >     > 
        >     >     >     
        >     >     >     
        >     >     >     
        >     >     >     
        >     >     >     
        >     >     >     ------------------------------------------------------------
        >     >     >     ------
        >     >     >     ---
        >     >     >     To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
        >     >     >     For additional commands, e-mail: dev-help@maven.apache.org
        >     >     >     
        >     >     >     
        >     >     > 
        >     >     > 
        >     >     > 
        >     >     > 
        >     >     > ----------------------------------------------------------------
        >     >     > -----
        >     >     > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
        >     >     > For additional commands, e-mail: dev-help@maven.apache.org
        >     >     > 
        >     > 
        >     > 
        >     > 
        >     >     
        >     >     
        >     >     
        >     >     
        >     >     
        >     >     ------------------------------------------------------------------
        >     >     ---
        >     >     To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
        >     >     For additional commands, e-mail: dev-help@maven.apache.org
        >     >     
        >     >     
        >     > 
        >     > 
        > 
        >     
        >     
        >     
        >     
        >     
        >     ---------------------------------------------------------------------
        >     To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
        >     For additional commands, e-mail: dev-help@maven.apache.org
        >     
        >     
        > 
        
        
        
        
        
        ---------------------------------------------------------------------
        To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
        For additional commands, e-mail: dev-help@maven.apache.org
        
        
    
    
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
    For additional commands, e-mail: dev-help@maven.apache.org
    
    
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
    For additional commands, e-mail: dev-help@maven.apache.org
    
    


Re: last review of Reproducible Builds proposal

Posted by he...@free.fr.
ok, Reproducible Builds are not so easy to get: each plugin that you use can cause an issue

I really recommend you diffoscope to investigate differences, it really helps a lot by easily giving you precise differences

Good luck for the investigations :)

Regards,

Hervé

----- Mail original -----
De: "Christofer Dutz" <ch...@c-ware.de>
À: "Maven Developers List" <de...@maven.apache.org>
Envoyé: Dimanche 3 Novembre 2019 20:16:42
Objet: Re: last review of Reproducible Builds proposal

Hi Herve,

unfortunately I now have implemented some tooling to compare the stuff produced by the updated reproducible builds.
And it does show quite a number of non binary equal files.

Will investigate what's the difference.

Chris



Am 03.11.19, 11:08 schrieb "Hervé BOUTEMY" <he...@free.fr>:

    great feedback, thank you: this proves the initial set of only 3 plugins is a 
    good basis.
    For checking the many output artifacts from a build, future buildinfo file [1] 
    should help a lot. I still have many usability concerns for Maven multi-module 
    builds (starting with: should we generate only the root buildinfo or one 
    buildinfo per Maven module?)
    Let's keep the ball rolling: today, it's plugins release day!!!
    
    Regards,
    
    Hervé
    
    [1] https://reproducible-builds.org/docs/jvm/
    
    Le vendredi 1 novembre 2019, 13:34:32 CET Christofer Dutz a écrit :
    > Hi,
    > 
    > so I just updated the versions of the 3 plugins and gave the Apache PLC4X
    > build a little spin ...
    > https://github.com/apache/plc4x/tree/feature/reproducible-builds
    > 
    > Did two executions of this: 
    > mvn -U -P
    > with-boost,with-java,with-dotnet,with-cpp,with-python,with-proxies,with-san
    > dbox,with
    > -DaltDeploymentRepository=snapshot-repo::default::file:./local-snapshots-di
    > r clean deploy
     (but with a different second local repo location)
    > 
    > Then I used "diff" and "cmp" to compare individual files and it didn't show
    > differences with the artifacts I chose ... 
     now I guess I'll have to whip
    > up some little bash script to sort of compare the artifacts from one
    > directory with the other (Unfortunately the SNAPSHOT timestamps are a
    > little annoying :-/ 
    > We do have some C++, C# and Python modules ... but I wouldn't expect the C++
    > and C# to be reproducible.
     
    > Chris
    > 
    > 
    > Am 01.11.19, 12:04 schrieb "Hervé BOUTEMY" <he...@free.fr>:
    > 
    >     Le vendredi 1 novembre 2019, 09:40:40 CET Christofer Dutz a écrit :
    > 
    >     > Hi Herve,
    >     > 
    >     > thanks for that will try it asap and report any findings back.
    >     > 
    >     > But good to know that there is a difference between JDK major versions
    >     > and
    >     > OSes ... so it would probably be best to stage releases on Linux with
    >     > an
    >     > OpenJDK of the minimum supported version?
    >     > Just thinking how to make it
    >     > possible to verify without having to buy Mac or Windows licenses ...
    >     > guess
    >     > on every machine you could whip up a Ubuntu VM for verification. Just
    >     > thinking about it ... perhaps it would be best to create a Docker
    >     > image for
     doing the reproducible stuff ...
    > 
    >     just to be clear: the difference is on newline only, then Windows vs
    > anything 
     else. You get the same result on Linux, MacOS, FreeBSD, or any
    > other Unix. 
    >     And if I want to be complete, if you get a source tarball from Windows,
    > 
     extract it to a Linux/MacOS/... box and build with "mvn -
    >     Dline.separator='\r\n'", in general, you get the same result as a build
    > under 
     Windows: I tested multiple times, it worked, but we'll see if it
    > works always or just "in general"
    >     
    > 
    >     > Are there any plans on creating a plugin to allow verification?
    >     > 
    >     > Sort of something like this:
    >     > "mvn package release:verify-reproduicble
    >     > -DstagingRepoUrl=a.b.c.de/repo/blahblahblah"
    >     > (Which doesn't deploy the
    >     > artifacts, but instead download them and do a binary comparison) 
    > 
    >     yes, but for now it was completely another form: this is the "Buildinfo
    > file" 
     proposal https://reproducible-builds.org/docs/jvm/
    >     that I stopped to work on 1 year ago given I had no chance to get the
    > same 
     output: it's now a good time to restart working on it as next steps 
    >     
    > 
    >     > Also it could be great if the release-plugin could automatically set
    >     > the
    >     > property:
    >     > a) if it finds the "project.build.outputTimestamp" set to some
    >     > placeholder value b) if some switch tells it to prepare a
    >     > reproducible
    >     > build by using some sort of "switch" parameter 
    >     > Guess that would sort of close the loop to get the biggest benefit out
    >     > of
    >     > the reproducible builds.
    > 
    >     +1
    >     issue has been created
    > https://issues.apache.org/jira/browse/MRELEASE-1029
     But I didn't work on
    > it yet, help welcome
    >     
    > 
    >     > I would be happy to help as I think this is one
    >     > of the greatest new features. (Ok ... perhaps besides the
    >     > sound-output-extension I learned about yesterday ;-) ) 
    > 
    >     +1
    >     the current step is important, but it's only the beginning of the story
    > from 
     an effective usage perspective
    >     
    >     Regards,
    >     
    >     Hervé
    >     
    > 
    >     > 
    >     > Chris
    >     > 
    >     > 
    >     > Am 01.11.19, 09:24 schrieb "Hervé BOUTEMY" <he...@free.fr>:
    >     > 
    >     > 
    >     >     Le jeudi 31 octobre 2019, 17:26:52 CET Christofer Dutz a écrit :
    >     > 
    >     > 
    >     > 
    >     >     > Hi all,
    >     >     > 
    >     >     > as I can see you're voting on releasing the reproducible build
    >     >     > extended
    >     >     > plugin versions.
    >     > 
    >     > 
    >     > 
    >     >      Is there any documentation on how to use this new
    >     > 
    >     > 
    >     > 
    >     >     > feature?
    >     >     > 
    >     >     > I had a look at the confluence page, but that seemed like a
    >     >     > brainstorming
    >     >     > session.
    >     > 
    >     > 
    >     > 
    >     >     ok, the Wiki page [1] started as a brainstorming session, was
    >     >     updated to
    >     > 
    >     > a proposal (the "Output Archive Entries Timestamp" parapgraph).
    > 
    >      And now I
    > 
    >     > probably should order paragraph a little bit, and add a "Making your
    >     > build
    >     > reproducible" section for end uses to have a quick explanation. 
    >     > 
    >     >     I'll write the explanation here as a first try before working on
    >     >     the
    >     > 
    >     > Wiki:
    > 
    >      
    > 
    >     >     1. upgrade your plugins to reproducible version, particularly
    >     > 
    >     > mpaven-source-plugin, maven-jar-plugin and maven-assembly-plugin to
    >     > vesion
    >     > 3.2.0
    > 
    >      2. add project.build.outputTimestamp property with the timestamp
    > 
    >     > value that will be used in zip/jar/tar archives: <properties>
    >     > 
    >     >        
    >     > 
    >     > <project.build.outputTimestamp>2019-10-02T08:04:00Z</project.build.out
    >     > putTi
     mestamp>
    > 
    >      </properties>
    > 
    >     >     
    >     >     Notice:
    >     >     - there is no Maven version prerequisite, everything happens at
    >     >     plugins
    >     > 
    >     > level
    > 
    >      - Reproducible Builds require to have no version ranges in
    > 
    >     > dependencies, generally gives different result on Unixes vs Windows,
    >     > and
    >     > generally depend on the major version of JDK used to compile (see "Out
    >     > of
    >     > Scope" paragraph) 
    >     > 
    >     >     You have the basis configured, the output should be reproducible
    >     >     now.
    >     >     If something is still not reproducible, use diffoscope to find
    >     >     the
    >     > 
    >     > unstable output, find the plugin that generated this output and check
    >     > if
    >     > there is a reproducible version available: if not, please open an
    >     > issue to
    >     > help plugin maintainers improving Reproducible Builds support at
    >     > every
    >     > plugin level.
    > 
    >      
    > 
    >     >     [1] 
    >     > 
    >     > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682
    >     > 318
    > 
    >      
    > 
    >     >     
    >     >     
    >     >      
    >     > 
    >     > 
    >     > 
    >     >     > I would love to add this to the PLC4X build asap.
    >     > 
    >     > 
    >     > 
    >     >     I'd love to have feedback
    >     >     Don't hesitate to ping me.
    >     > 
    >     > 
    >     > 
    >     >     > 
    >     >     > So I would like to test the release-candidates and vote too.
    >     > 
    >     > 
    >     > 
    >     >     I would love to have many tester and votes :)
    >     >     
    >     > 
    >     > 
    >     > 
    >     >     > 
    >     >     > Chris
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > Am 16.10.19, 14:42 schrieb "Hervé BOUTEMY"
    >     >     > <he...@free.fr>:
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     >     Le mercredi 16 octobre 2019, 13:40:48 CEST Andreas Sewe a
    >     >     >     écrit :
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     >     > Emmanuel Bourg wrote:
    >     >     >     > 
    >     >     >     > 
    >     >     >     > 
    >     >     >     > > Le 16/10/2019 à 08:35, Hervé BOUTEMY a écrit :
    >     >     >     > > 
    >     >     >     > > 
    >     >     >     > > 
    >     >     >     > >> last question: now that we seem to fully understand
    >     >     >     > >> each
    >     >     >     > >> other,
    >     >     >     > >> does it
    >     >     >     > >> mean that you don't need any more "seconds since the
    >     >     >     > >> epoch"
    >     >     >     > >> format
    >     >     >     > >> support for the property?
    >     >     >     > > 
    >     >     >     > > 
    >     >     >     > > 
    >     >     >     > > 
    >     >     >     > > If Maven supports the SOURCE_DATE_EPOCH environment
    >     >     >     > > variable
    >     >     >     > > I
    >     >     >     > > don't
    >     >     >     > > think that's necessary, otherwise it would be nice to be
    >     >     >     > > able
    >     >     >     > > to
    >     >     >     > > invoke
    >     >     >     > > 
    >     >     >     > > Maven with:
    >     >     >     > > 
    >     >     >     > > 
    >     >     >     > > 
    >     >     >     > >    mvn package
    >     >     >     > >    -Dproject.build.outputTimestamp=$SOURCE_DATE_EPOCH
    >     >     >     > > 
    >     >     >     > > 
    >     >     >     > > 
    >     >     >     > > 
    >     >     >     > > and this means supporting a timestamp formatted as
    >     >     >     > > seconds
    >     >     >     > > since
    >     >     >     > > the
    >     >     >     > > epoch.
    >     >     >     > 
    >     >     >     > 
    >     >     >     > 
    >     >     >     > 
    >     >     >     > +1
    >     >     >     > 
    >     >     >     > The above would be a nice, simple way of bridging the gap
    >     >     >     > between
    >     >     >     > SOURCE_DATE_EPOCH and project.build.outputTimestamp.
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     >     told like that, ok, why not
    >     >     >     
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     >     > 
    >     >     >     > If it is not too much trouble to implement the "\d+ ->
    >     >     >     > seconds
    >     >     >     > since
    >     >     >     > epoch" heuristic, them I would love to see it included.
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     >     ok, I'll do and prepare the release
    >     >     >     
    >     >     >     Regards,
    >     >     >     
    >     >     >     Hervé
    >     >     >     
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     >     > 
    >     >     >     > Best wishes,
    >     >     >     > 
    >     >     >     > Andreas
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     >     
    >     >     >     
    >     >     >     
    >     >     >     
    >     >     >     
    >     >     >     ------------------------------------------------------------
    >     >     >     ------
    >     >     >     ---
    >     >     >     To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
    >     >     >     For additional commands, e-mail: dev-help@maven.apache.org
    >     >     >     
    >     >     >     
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > ----------------------------------------------------------------
    >     >     > -----
    >     >     > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
    >     >     > For additional commands, e-mail: dev-help@maven.apache.org
    >     >     > 
    >     > 
    >     > 
    >     > 
    >     >     
    >     >     
    >     >     
    >     >     
    >     >     
    >     >     ------------------------------------------------------------------
    >     >     ---
    >     >     To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
    >     >     For additional commands, e-mail: dev-help@maven.apache.org
    >     >     
    >     >     
    >     > 
    >     > 
    > 
    >     
    >     
    >     
    >     
    >     
    >     ---------------------------------------------------------------------
    >     To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
    >     For additional commands, e-mail: dev-help@maven.apache.org
    >     
    >     
    > 
    
    
    
    
    
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
    For additional commands, e-mail: dev-help@maven.apache.org
    
    


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Re: last review of Reproducible Builds proposal

Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Herve,

unfortunately I now have implemented some tooling to compare the stuff produced by the updated reproducible builds.
And it does show quite a number of non binary equal files.

Will investigate what's the difference.

Chris



Am 03.11.19, 11:08 schrieb "Hervé BOUTEMY" <he...@free.fr>:

    great feedback, thank you: this proves the initial set of only 3 plugins is a 
    good basis.
    For checking the many output artifacts from a build, future buildinfo file [1] 
    should help a lot. I still have many usability concerns for Maven multi-module 
    builds (starting with: should we generate only the root buildinfo or one 
    buildinfo per Maven module?)
    Let's keep the ball rolling: today, it's plugins release day!!!
    
    Regards,
    
    Hervé
    
    [1] https://reproducible-builds.org/docs/jvm/
    
    Le vendredi 1 novembre 2019, 13:34:32 CET Christofer Dutz a écrit :
    > Hi,
    > 
    > so I just updated the versions of the 3 plugins and gave the Apache PLC4X
    > build a little spin ...
    > https://github.com/apache/plc4x/tree/feature/reproducible-builds
    > 
    > Did two executions of this: 
    > mvn -U -P
    > with-boost,with-java,with-dotnet,with-cpp,with-python,with-proxies,with-san
    > dbox,with
    > -DaltDeploymentRepository=snapshot-repo::default::file:./local-snapshots-di
    > r clean deploy
     (but with a different second local repo location)
    > 
    > Then I used "diff" and "cmp" to compare individual files and it didn't show
    > differences with the artifacts I chose ... 
     now I guess I'll have to whip
    > up some little bash script to sort of compare the artifacts from one
    > directory with the other (Unfortunately the SNAPSHOT timestamps are a
    > little annoying :-/ 
    > We do have some C++, C# and Python modules ... but I wouldn't expect the C++
    > and C# to be reproducible.
     
    > Chris
    > 
    > 
    > Am 01.11.19, 12:04 schrieb "Hervé BOUTEMY" <he...@free.fr>:
    > 
    >     Le vendredi 1 novembre 2019, 09:40:40 CET Christofer Dutz a écrit :
    > 
    >     > Hi Herve,
    >     > 
    >     > thanks for that will try it asap and report any findings back.
    >     > 
    >     > But good to know that there is a difference between JDK major versions
    >     > and
    >     > OSes ... so it would probably be best to stage releases on Linux with
    >     > an
    >     > OpenJDK of the minimum supported version?
    >     > Just thinking how to make it
    >     > possible to verify without having to buy Mac or Windows licenses ...
    >     > guess
    >     > on every machine you could whip up a Ubuntu VM for verification. Just
    >     > thinking about it ... perhaps it would be best to create a Docker
    >     > image for
     doing the reproducible stuff ...
    > 
    >     just to be clear: the difference is on newline only, then Windows vs
    > anything 
     else. You get the same result on Linux, MacOS, FreeBSD, or any
    > other Unix. 
    >     And if I want to be complete, if you get a source tarball from Windows,
    > 
     extract it to a Linux/MacOS/... box and build with "mvn -
    >     Dline.separator='\r\n'", in general, you get the same result as a build
    > under 
     Windows: I tested multiple times, it worked, but we'll see if it
    > works always or just "in general"
    >     
    > 
    >     > Are there any plans on creating a plugin to allow verification?
    >     > 
    >     > Sort of something like this:
    >     > "mvn package release:verify-reproduicble
    >     > -DstagingRepoUrl=a.b.c.de/repo/blahblahblah"
    >     > (Which doesn't deploy the
    >     > artifacts, but instead download them and do a binary comparison) 
    > 
    >     yes, but for now it was completely another form: this is the "Buildinfo
    > file" 
     proposal https://reproducible-builds.org/docs/jvm/
    >     that I stopped to work on 1 year ago given I had no chance to get the
    > same 
     output: it's now a good time to restart working on it as next steps 
    >     
    > 
    >     > Also it could be great if the release-plugin could automatically set
    >     > the
    >     > property:
    >     > a) if it finds the "project.build.outputTimestamp" set to some
    >     > placeholder value b) if some switch tells it to prepare a
    >     > reproducible
    >     > build by using some sort of "switch" parameter 
    >     > Guess that would sort of close the loop to get the biggest benefit out
    >     > of
    >     > the reproducible builds.
    > 
    >     +1
    >     issue has been created
    > https://issues.apache.org/jira/browse/MRELEASE-1029
     But I didn't work on
    > it yet, help welcome
    >     
    > 
    >     > I would be happy to help as I think this is one
    >     > of the greatest new features. (Ok ... perhaps besides the
    >     > sound-output-extension I learned about yesterday ;-) ) 
    > 
    >     +1
    >     the current step is important, but it's only the beginning of the story
    > from 
     an effective usage perspective
    >     
    >     Regards,
    >     
    >     Hervé
    >     
    > 
    >     > 
    >     > Chris
    >     > 
    >     > 
    >     > Am 01.11.19, 09:24 schrieb "Hervé BOUTEMY" <he...@free.fr>:
    >     > 
    >     > 
    >     >     Le jeudi 31 octobre 2019, 17:26:52 CET Christofer Dutz a écrit :
    >     > 
    >     > 
    >     > 
    >     >     > Hi all,
    >     >     > 
    >     >     > as I can see you're voting on releasing the reproducible build
    >     >     > extended
    >     >     > plugin versions.
    >     > 
    >     > 
    >     > 
    >     >      Is there any documentation on how to use this new
    >     > 
    >     > 
    >     > 
    >     >     > feature?
    >     >     > 
    >     >     > I had a look at the confluence page, but that seemed like a
    >     >     > brainstorming
    >     >     > session.
    >     > 
    >     > 
    >     > 
    >     >     ok, the Wiki page [1] started as a brainstorming session, was
    >     >     updated to
    >     > 
    >     > a proposal (the "Output Archive Entries Timestamp" parapgraph).
    > 
    >      And now I
    > 
    >     > probably should order paragraph a little bit, and add a "Making your
    >     > build
    >     > reproducible" section for end uses to have a quick explanation. 
    >     > 
    >     >     I'll write the explanation here as a first try before working on
    >     >     the
    >     > 
    >     > Wiki:
    > 
    >      
    > 
    >     >     1. upgrade your plugins to reproducible version, particularly
    >     > 
    >     > mpaven-source-plugin, maven-jar-plugin and maven-assembly-plugin to
    >     > vesion
    >     > 3.2.0
    > 
    >      2. add project.build.outputTimestamp property with the timestamp
    > 
    >     > value that will be used in zip/jar/tar archives: <properties>
    >     > 
    >     >        
    >     > 
    >     > <project.build.outputTimestamp>2019-10-02T08:04:00Z</project.build.out
    >     > putTi
     mestamp>
    > 
    >      </properties>
    > 
    >     >     
    >     >     Notice:
    >     >     - there is no Maven version prerequisite, everything happens at
    >     >     plugins
    >     > 
    >     > level
    > 
    >      - Reproducible Builds require to have no version ranges in
    > 
    >     > dependencies, generally gives different result on Unixes vs Windows,
    >     > and
    >     > generally depend on the major version of JDK used to compile (see "Out
    >     > of
    >     > Scope" paragraph) 
    >     > 
    >     >     You have the basis configured, the output should be reproducible
    >     >     now.
    >     >     If something is still not reproducible, use diffoscope to find
    >     >     the
    >     > 
    >     > unstable output, find the plugin that generated this output and check
    >     > if
    >     > there is a reproducible version available: if not, please open an
    >     > issue to
    >     > help plugin maintainers improving Reproducible Builds support at
    >     > every
    >     > plugin level.
    > 
    >      
    > 
    >     >     [1] 
    >     > 
    >     > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682
    >     > 318
    > 
    >      
    > 
    >     >     
    >     >     
    >     >      
    >     > 
    >     > 
    >     > 
    >     >     > I would love to add this to the PLC4X build asap.
    >     > 
    >     > 
    >     > 
    >     >     I'd love to have feedback
    >     >     Don't hesitate to ping me.
    >     > 
    >     > 
    >     > 
    >     >     > 
    >     >     > So I would like to test the release-candidates and vote too.
    >     > 
    >     > 
    >     > 
    >     >     I would love to have many tester and votes :)
    >     >     
    >     > 
    >     > 
    >     > 
    >     >     > 
    >     >     > Chris
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > Am 16.10.19, 14:42 schrieb "Hervé BOUTEMY"
    >     >     > <he...@free.fr>:
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     >     Le mercredi 16 octobre 2019, 13:40:48 CEST Andreas Sewe a
    >     >     >     écrit :
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     >     > Emmanuel Bourg wrote:
    >     >     >     > 
    >     >     >     > 
    >     >     >     > 
    >     >     >     > > Le 16/10/2019 à 08:35, Hervé BOUTEMY a écrit :
    >     >     >     > > 
    >     >     >     > > 
    >     >     >     > > 
    >     >     >     > >> last question: now that we seem to fully understand
    >     >     >     > >> each
    >     >     >     > >> other,
    >     >     >     > >> does it
    >     >     >     > >> mean that you don't need any more "seconds since the
    >     >     >     > >> epoch"
    >     >     >     > >> format
    >     >     >     > >> support for the property?
    >     >     >     > > 
    >     >     >     > > 
    >     >     >     > > 
    >     >     >     > > 
    >     >     >     > > If Maven supports the SOURCE_DATE_EPOCH environment
    >     >     >     > > variable
    >     >     >     > > I
    >     >     >     > > don't
    >     >     >     > > think that's necessary, otherwise it would be nice to be
    >     >     >     > > able
    >     >     >     > > to
    >     >     >     > > invoke
    >     >     >     > > 
    >     >     >     > > Maven with:
    >     >     >     > > 
    >     >     >     > > 
    >     >     >     > > 
    >     >     >     > >    mvn package
    >     >     >     > >    -Dproject.build.outputTimestamp=$SOURCE_DATE_EPOCH
    >     >     >     > > 
    >     >     >     > > 
    >     >     >     > > 
    >     >     >     > > 
    >     >     >     > > and this means supporting a timestamp formatted as
    >     >     >     > > seconds
    >     >     >     > > since
    >     >     >     > > the
    >     >     >     > > epoch.
    >     >     >     > 
    >     >     >     > 
    >     >     >     > 
    >     >     >     > 
    >     >     >     > +1
    >     >     >     > 
    >     >     >     > The above would be a nice, simple way of bridging the gap
    >     >     >     > between
    >     >     >     > SOURCE_DATE_EPOCH and project.build.outputTimestamp.
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     >     told like that, ok, why not
    >     >     >     
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     >     > 
    >     >     >     > If it is not too much trouble to implement the "\d+ ->
    >     >     >     > seconds
    >     >     >     > since
    >     >     >     > epoch" heuristic, them I would love to see it included.
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     >     ok, I'll do and prepare the release
    >     >     >     
    >     >     >     Regards,
    >     >     >     
    >     >     >     Hervé
    >     >     >     
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     >     > 
    >     >     >     > Best wishes,
    >     >     >     > 
    >     >     >     > Andreas
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     >     
    >     >     >     
    >     >     >     
    >     >     >     
    >     >     >     
    >     >     >     ------------------------------------------------------------
    >     >     >     ------
    >     >     >     ---
    >     >     >     To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
    >     >     >     For additional commands, e-mail: dev-help@maven.apache.org
    >     >     >     
    >     >     >     
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > ----------------------------------------------------------------
    >     >     > -----
    >     >     > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
    >     >     > For additional commands, e-mail: dev-help@maven.apache.org
    >     >     > 
    >     > 
    >     > 
    >     > 
    >     >     
    >     >     
    >     >     
    >     >     
    >     >     
    >     >     ------------------------------------------------------------------
    >     >     ---
    >     >     To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
    >     >     For additional commands, e-mail: dev-help@maven.apache.org
    >     >     
    >     >     
    >     > 
    >     > 
    > 
    >     
    >     
    >     
    >     
    >     
    >     ---------------------------------------------------------------------
    >     To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
    >     For additional commands, e-mail: dev-help@maven.apache.org
    >     
    >     
    > 
    
    
    
    
    
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
    For additional commands, e-mail: dev-help@maven.apache.org
    
    


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Re: last review of Reproducible Builds proposal

Posted by Hervé BOUTEMY <he...@free.fr>.
great feedback, thank you: this proves the initial set of only 3 plugins is a 
good basis.
For checking the many output artifacts from a build, future buildinfo file [1] 
should help a lot. I still have many usability concerns for Maven multi-module 
builds (starting with: should we generate only the root buildinfo or one 
buildinfo per Maven module?)
Let's keep the ball rolling: today, it's plugins release day!!!

Regards,

Hervé

[1] https://reproducible-builds.org/docs/jvm/

Le vendredi 1 novembre 2019, 13:34:32 CET Christofer Dutz a écrit :
> Hi,
> 
> so I just updated the versions of the 3 plugins and gave the Apache PLC4X
> build a little spin ...
> https://github.com/apache/plc4x/tree/feature/reproducible-builds
> 
> Did two executions of this: 
> mvn -U -P
> with-boost,with-java,with-dotnet,with-cpp,with-python,with-proxies,with-san
> dbox,with
> -DaltDeploymentRepository=snapshot-repo::default::file:./local-snapshots-di
> r clean deploy
 (but with a different second local repo location)
> 
> Then I used "diff" and "cmp" to compare individual files and it didn't show
> differences with the artifacts I chose ... 
 now I guess I'll have to whip
> up some little bash script to sort of compare the artifacts from one
> directory with the other (Unfortunately the SNAPSHOT timestamps are a
> little annoying :-/ 
> We do have some C++, C# and Python modules ... but I wouldn't expect the C++
> and C# to be reproducible.
 
> Chris
> 
> 
> Am 01.11.19, 12:04 schrieb "Hervé BOUTEMY" <he...@free.fr>:
> 
>     Le vendredi 1 novembre 2019, 09:40:40 CET Christofer Dutz a écrit :
> 
>     > Hi Herve,
>     > 
>     > thanks for that will try it asap and report any findings back.
>     > 
>     > But good to know that there is a difference between JDK major versions
>     > and
>     > OSes ... so it would probably be best to stage releases on Linux with
>     > an
>     > OpenJDK of the minimum supported version?
>     > Just thinking how to make it
>     > possible to verify without having to buy Mac or Windows licenses ...
>     > guess
>     > on every machine you could whip up a Ubuntu VM for verification. Just
>     > thinking about it ... perhaps it would be best to create a Docker
>     > image for
 doing the reproducible stuff ...
> 
>     just to be clear: the difference is on newline only, then Windows vs
> anything 
 else. You get the same result on Linux, MacOS, FreeBSD, or any
> other Unix. 
>     And if I want to be complete, if you get a source tarball from Windows,
> 
 extract it to a Linux/MacOS/... box and build with "mvn -
>     Dline.separator='\r\n'", in general, you get the same result as a build
> under 
 Windows: I tested multiple times, it worked, but we'll see if it
> works always or just "in general"
>     
> 
>     > Are there any plans on creating a plugin to allow verification?
>     > 
>     > Sort of something like this:
>     > "mvn package release:verify-reproduicble
>     > -DstagingRepoUrl=a.b.c.de/repo/blahblahblah"
>     > (Which doesn't deploy the
>     > artifacts, but instead download them and do a binary comparison) 
> 
>     yes, but for now it was completely another form: this is the "Buildinfo
> file" 
 proposal https://reproducible-builds.org/docs/jvm/
>     that I stopped to work on 1 year ago given I had no chance to get the
> same 
 output: it's now a good time to restart working on it as next steps 
>     
> 
>     > Also it could be great if the release-plugin could automatically set
>     > the
>     > property:
>     > a) if it finds the "project.build.outputTimestamp" set to some
>     > placeholder value b) if some switch tells it to prepare a
>     > reproducible
>     > build by using some sort of "switch" parameter 
>     > Guess that would sort of close the loop to get the biggest benefit out
>     > of
>     > the reproducible builds.
> 
>     +1
>     issue has been created
> https://issues.apache.org/jira/browse/MRELEASE-1029
 But I didn't work on
> it yet, help welcome
>     
> 
>     > I would be happy to help as I think this is one
>     > of the greatest new features. (Ok ... perhaps besides the
>     > sound-output-extension I learned about yesterday ;-) ) 
> 
>     +1
>     the current step is important, but it's only the beginning of the story
> from 
 an effective usage perspective
>     
>     Regards,
>     
>     Hervé
>     
> 
>     > 
>     > Chris
>     > 
>     > 
>     > Am 01.11.19, 09:24 schrieb "Hervé BOUTEMY" <he...@free.fr>:
>     > 
>     > 
>     >     Le jeudi 31 octobre 2019, 17:26:52 CET Christofer Dutz a écrit :
>     > 
>     > 
>     > 
>     >     > Hi all,
>     >     > 
>     >     > as I can see you're voting on releasing the reproducible build
>     >     > extended
>     >     > plugin versions.
>     > 
>     > 
>     > 
>     >      Is there any documentation on how to use this new
>     > 
>     > 
>     > 
>     >     > feature?
>     >     > 
>     >     > I had a look at the confluence page, but that seemed like a
>     >     > brainstorming
>     >     > session.
>     > 
>     > 
>     > 
>     >     ok, the Wiki page [1] started as a brainstorming session, was
>     >     updated to
>     > 
>     > a proposal (the "Output Archive Entries Timestamp" parapgraph).
> 
>      And now I
> 
>     > probably should order paragraph a little bit, and add a "Making your
>     > build
>     > reproducible" section for end uses to have a quick explanation. 
>     > 
>     >     I'll write the explanation here as a first try before working on
>     >     the
>     > 
>     > Wiki:
> 
>      
> 
>     >     1. upgrade your plugins to reproducible version, particularly
>     > 
>     > mpaven-source-plugin, maven-jar-plugin and maven-assembly-plugin to
>     > vesion
>     > 3.2.0
> 
>      2. add project.build.outputTimestamp property with the timestamp
> 
>     > value that will be used in zip/jar/tar archives: <properties>
>     > 
>     >        
>     > 
>     > <project.build.outputTimestamp>2019-10-02T08:04:00Z</project.build.out
>     > putTi
 mestamp>
> 
>      </properties>
> 
>     >     
>     >     Notice:
>     >     - there is no Maven version prerequisite, everything happens at
>     >     plugins
>     > 
>     > level
> 
>      - Reproducible Builds require to have no version ranges in
> 
>     > dependencies, generally gives different result on Unixes vs Windows,
>     > and
>     > generally depend on the major version of JDK used to compile (see "Out
>     > of
>     > Scope" paragraph) 
>     > 
>     >     You have the basis configured, the output should be reproducible
>     >     now.
>     >     If something is still not reproducible, use diffoscope to find
>     >     the
>     > 
>     > unstable output, find the plugin that generated this output and check
>     > if
>     > there is a reproducible version available: if not, please open an
>     > issue to
>     > help plugin maintainers improving Reproducible Builds support at
>     > every
>     > plugin level.
> 
>      
> 
>     >     [1] 
>     > 
>     > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682
>     > 318
> 
>      
> 
>     >     
>     >     
>     >      
>     > 
>     > 
>     > 
>     >     > I would love to add this to the PLC4X build asap.
>     > 
>     > 
>     > 
>     >     I'd love to have feedback
>     >     Don't hesitate to ping me.
>     > 
>     > 
>     > 
>     >     > 
>     >     > So I would like to test the release-candidates and vote too.
>     > 
>     > 
>     > 
>     >     I would love to have many tester and votes :)
>     >     
>     > 
>     > 
>     > 
>     >     > 
>     >     > Chris
>     >     > 
>     >     > 
>     >     > 
>     >     > Am 16.10.19, 14:42 schrieb "Hervé BOUTEMY"
>     >     > <he...@free.fr>:
>     >     > 
>     >     > 
>     >     > 
>     >     >     Le mercredi 16 octobre 2019, 13:40:48 CEST Andreas Sewe a
>     >     >     écrit :
>     >     > 
>     >     > 
>     >     > 
>     >     > 
>     >     > 
>     >     >     > Emmanuel Bourg wrote:
>     >     >     > 
>     >     >     > 
>     >     >     > 
>     >     >     > > Le 16/10/2019 à 08:35, Hervé BOUTEMY a écrit :
>     >     >     > > 
>     >     >     > > 
>     >     >     > > 
>     >     >     > >> last question: now that we seem to fully understand
>     >     >     > >> each
>     >     >     > >> other,
>     >     >     > >> does it
>     >     >     > >> mean that you don't need any more "seconds since the
>     >     >     > >> epoch"
>     >     >     > >> format
>     >     >     > >> support for the property?
>     >     >     > > 
>     >     >     > > 
>     >     >     > > 
>     >     >     > > 
>     >     >     > > If Maven supports the SOURCE_DATE_EPOCH environment
>     >     >     > > variable
>     >     >     > > I
>     >     >     > > don't
>     >     >     > > think that's necessary, otherwise it would be nice to be
>     >     >     > > able
>     >     >     > > to
>     >     >     > > invoke
>     >     >     > > 
>     >     >     > > Maven with:
>     >     >     > > 
>     >     >     > > 
>     >     >     > > 
>     >     >     > >    mvn package
>     >     >     > >    -Dproject.build.outputTimestamp=$SOURCE_DATE_EPOCH
>     >     >     > > 
>     >     >     > > 
>     >     >     > > 
>     >     >     > > 
>     >     >     > > and this means supporting a timestamp formatted as
>     >     >     > > seconds
>     >     >     > > since
>     >     >     > > the
>     >     >     > > epoch.
>     >     >     > 
>     >     >     > 
>     >     >     > 
>     >     >     > 
>     >     >     > +1
>     >     >     > 
>     >     >     > The above would be a nice, simple way of bridging the gap
>     >     >     > between
>     >     >     > SOURCE_DATE_EPOCH and project.build.outputTimestamp.
>     >     > 
>     >     > 
>     >     > 
>     >     > 
>     >     > 
>     >     >     told like that, ok, why not
>     >     >     
>     >     > 
>     >     > 
>     >     > 
>     >     > 
>     >     > 
>     >     >     > 
>     >     >     > If it is not too much trouble to implement the "\d+ ->
>     >     >     > seconds
>     >     >     > since
>     >     >     > epoch" heuristic, them I would love to see it included.
>     >     > 
>     >     > 
>     >     > 
>     >     > 
>     >     > 
>     >     >     ok, I'll do and prepare the release
>     >     >     
>     >     >     Regards,
>     >     >     
>     >     >     Hervé
>     >     >     
>     >     > 
>     >     > 
>     >     > 
>     >     > 
>     >     > 
>     >     >     > 
>     >     >     > Best wishes,
>     >     >     > 
>     >     >     > Andreas
>     >     > 
>     >     > 
>     >     > 
>     >     > 
>     >     > 
>     >     >     
>     >     >     
>     >     >     
>     >     >     
>     >     >     
>     >     >     ------------------------------------------------------------
>     >     >     ------
>     >     >     ---
>     >     >     To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>     >     >     For additional commands, e-mail: dev-help@maven.apache.org
>     >     >     
>     >     >     
>     >     > 
>     >     > 
>     >     > 
>     >     > 
>     >     > ----------------------------------------------------------------
>     >     > -----
>     >     > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>     >     > For additional commands, e-mail: dev-help@maven.apache.org
>     >     > 
>     > 
>     > 
>     > 
>     >     
>     >     
>     >     
>     >     
>     >     
>     >     ------------------------------------------------------------------
>     >     ---
>     >     To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>     >     For additional commands, e-mail: dev-help@maven.apache.org
>     >     
>     >     
>     > 
>     > 
> 
>     
>     
>     
>     
>     
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>     For additional commands, e-mail: dev-help@maven.apache.org
>     
>     
> 





---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Re: last review of Reproducible Builds proposal

Posted by Christofer Dutz <ch...@c-ware.de>.
Hi,

so I just updated the versions of the 3 plugins and gave the Apache PLC4X build a little spin ...
https://github.com/apache/plc4x/tree/feature/reproducible-builds

Did two executions of this: 
mvn -U -P with-boost,with-java,with-dotnet,with-cpp,with-python,with-proxies,with-sandbox,with -DaltDeploymentRepository=snapshot-repo::default::file:./local-snapshots-dir clean deploy
 (but with a different second local repo location)

Then I used "diff" and "cmp" to compare individual files and it didn't show differences with the artifacts I chose ... 
now I guess I'll have to whip up some little bash script to sort of compare the artifacts from one directory with the other (Unfortunately the SNAPSHOT timestamps are a little annoying :-/

We do have some C++, C# and Python modules ... but I wouldn't expect the C++ and C# to be reproducible.

Chris


Am 01.11.19, 12:04 schrieb "Hervé BOUTEMY" <he...@free.fr>:

    Le vendredi 1 novembre 2019, 09:40:40 CET Christofer Dutz a écrit :
    > Hi Herve,
    > 
    > thanks for that will try it asap and report any findings back.
    > 
    > But good to know that there is a difference between JDK major versions and
    > OSes ... so it would probably be best to stage releases on Linux with an
    > OpenJDK of the minimum supported version?
    > Just thinking how to make it
    > possible to verify without having to buy Mac or Windows licenses ... guess
    > on every machine you could whip up a Ubuntu VM for verification. Just
    > thinking about it ... perhaps it would be best to create a Docker image for
    > doing the reproducible stuff ... 
    just to be clear: the difference is on newline only, then Windows vs anything 
    else. You get the same result on Linux, MacOS, FreeBSD, or any other Unix.
    
    And if I want to be complete, if you get a source tarball from Windows, 
    extract it to a Linux/MacOS/... box and build with "mvn -
    Dline.separator='\r\n'", in general, you get the same result as a build under 
    Windows: I tested multiple times, it worked, but we'll see if it works always 
    or just "in general"
    
    > Are there any plans on creating a plugin to allow verification?
    > 
    > Sort of something like this:
    > "mvn package release:verify-reproduicble
    > -DstagingRepoUrl=a.b.c.de/repo/blahblahblah"
    > (Which doesn't deploy the
    > artifacts, but instead download them and do a binary comparison) 
    yes, but for now it was completely another form: this is the "Buildinfo file" 
    proposal https://reproducible-builds.org/docs/jvm/
    that I stopped to work on 1 year ago given I had no chance to get the same 
    output: it's now a good time to restart working on it as next steps
    
    
    > Also it could be great if the release-plugin could automatically set the
    > property:
    > a) if it finds the "project.build.outputTimestamp" set to some
    > placeholder value b) if some switch tells it to prepare a reproducible
    > build by using some sort of "switch" parameter 
    > Guess that would sort of close the loop to get the biggest benefit out of
    > the reproducible builds.
    +1
    issue has been created https://issues.apache.org/jira/browse/MRELEASE-1029
    But I didn't work on it yet, help welcome
    
    > I would be happy to help as I think this is one
    > of the greatest new features. (Ok ... perhaps besides the
    > sound-output-extension I learned about yesterday ;-) ) 
    +1
    the current step is important, but it's only the beginning of the story from 
    an effective usage perspective
    
    Regards,
    
    Hervé
    
    > 
    > Chris
    > 
    > 
    > Am 01.11.19, 09:24 schrieb "Hervé BOUTEMY" <he...@free.fr>:
    > 
    >     Le jeudi 31 octobre 2019, 17:26:52 CET Christofer Dutz a écrit :
    > 
    >     > Hi all,
    >     > 
    >     > as I can see you're voting on releasing the reproducible build
    >     > extended
    >     > plugin versions.
    > 
    >      Is there any documentation on how to use this new
    > 
    >     > feature?
    >     > 
    >     > I had a look at the confluence page, but that seemed like a
    >     > brainstorming
    >     > session.
    > 
    >     ok, the Wiki page [1] started as a brainstorming session, was updated to
    > a proposal (the "Output Archive Entries Timestamp" parapgraph).
     And now I
    > probably should order paragraph a little bit, and add a "Making your build
    > reproducible" section for end uses to have a quick explanation. 
    >     I'll write the explanation here as a first try before working on the
    > Wiki:
     
    >     1. upgrade your plugins to reproducible version, particularly
    > mpaven-source-plugin, maven-jar-plugin and maven-assembly-plugin to vesion
    > 3.2.0
     2. add project.build.outputTimestamp property with the timestamp
    > value that will be used in zip/jar/tar archives: <properties>
    >        
    > <project.build.outputTimestamp>2019-10-02T08:04:00Z</project.build.outputTi
    > mestamp>
     </properties>
    >     
    >     Notice:
    >     - there is no Maven version prerequisite, everything happens at plugins
    > level
     - Reproducible Builds require to have no version ranges in
    > dependencies, generally gives different result on Unixes vs Windows, and
    > generally depend on the major version of JDK used to compile (see "Out of
    > Scope" paragraph) 
    >     You have the basis configured, the output should be reproducible now.
    >     If something is still not reproducible, use diffoscope to find the
    > unstable output, find the plugin that generated this output and check if
    > there is a reproducible version available: if not, please open an issue to
    > help plugin maintainers improving Reproducible Builds support at every
    > plugin level.
     
    >     [1] 
    > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318
     
    >     
    >      
    > 
    >     > I would love to add this to the PLC4X build asap.
    > 
    >     I'd love to have feedback
    >     Don't hesitate to ping me.
    > 
    >     > 
    >     > So I would like to test the release-candidates and vote too.
    > 
    >     I would love to have many tester and votes :)
    >     
    > 
    >     > 
    >     > Chris
    >     > 
    >     > 
    >     > 
    >     > Am 16.10.19, 14:42 schrieb "Hervé BOUTEMY" <he...@free.fr>:
    >     > 
    >     > 
    >     >     Le mercredi 16 octobre 2019, 13:40:48 CEST Andreas Sewe a écrit :
    >     > 
    >     > 
    >     > 
    >     >     > Emmanuel Bourg wrote:
    >     >     > 
    >     >     > 
    >     >     > > Le 16/10/2019 à 08:35, Hervé BOUTEMY a écrit :
    >     >     > > 
    >     >     > > 
    >     >     > >> last question: now that we seem to fully understand each
    >     >     > >> other,
    >     >     > >> does it
    >     >     > >> mean that you don't need any more "seconds since the epoch"
    >     >     > >> format
    >     >     > >> support for the property?
    >     >     > > 
    >     >     > > 
    >     >     > > 
    >     >     > > If Maven supports the SOURCE_DATE_EPOCH environment variable
    >     >     > > I
    >     >     > > don't
    >     >     > > think that's necessary, otherwise it would be nice to be able
    >     >     > > to
    >     >     > > invoke
    >     >     > > 
    >     >     > > Maven with:
    >     >     > > 
    >     >     > > 
    >     >     > >    mvn package
    >     >     > >    -Dproject.build.outputTimestamp=$SOURCE_DATE_EPOCH
    >     >     > > 
    >     >     > > 
    >     >     > > 
    >     >     > > and this means supporting a timestamp formatted as seconds
    >     >     > > since
    >     >     > > the
    >     >     > > epoch.
    >     >     > 
    >     >     > 
    >     >     > 
    >     >     > +1
    >     >     > 
    >     >     > The above would be a nice, simple way of bridging the gap
    >     >     > between
    >     >     > SOURCE_DATE_EPOCH and project.build.outputTimestamp.
    >     > 
    >     > 
    >     > 
    >     >     told like that, ok, why not
    >     >     
    >     > 
    >     > 
    >     > 
    >     >     > 
    >     >     > If it is not too much trouble to implement the "\d+ -> seconds
    >     >     > since
    >     >     > epoch" heuristic, them I would love to see it included.
    >     > 
    >     > 
    >     > 
    >     >     ok, I'll do and prepare the release
    >     >     
    >     >     Regards,
    >     >     
    >     >     Hervé
    >     >     
    >     > 
    >     > 
    >     > 
    >     >     > 
    >     >     > Best wishes,
    >     >     > 
    >     >     > Andreas
    >     > 
    >     > 
    >     > 
    >     >     
    >     >     
    >     >     
    >     >     
    >     >     
    >     >     ------------------------------------------------------------------
    >     >     ---
    >     >     To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
    >     >     For additional commands, e-mail: dev-help@maven.apache.org
    >     >     
    >     >     
    >     > 
    >     > 
    >     > 
    >     > ---------------------------------------------------------------------
    >     > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
    >     > For additional commands, e-mail: dev-help@maven.apache.org
    >     > 
    > 
    >     
    >     
    >     
    >     
    >     
    >     ---------------------------------------------------------------------
    >     To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
    >     For additional commands, e-mail: dev-help@maven.apache.org
    >     
    >     
    > 
    
    
    
    
    
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
    For additional commands, e-mail: dev-help@maven.apache.org
    
    


Re: last review of Reproducible Builds proposal

Posted by Hervé BOUTEMY <he...@free.fr>.
Le vendredi 1 novembre 2019, 09:40:40 CET Christofer Dutz a écrit :
> Hi Herve,
> 
> thanks for that will try it asap and report any findings back.
> 
> But good to know that there is a difference between JDK major versions and
> OSes ... so it would probably be best to stage releases on Linux with an
> OpenJDK of the minimum supported version?
> Just thinking how to make it
> possible to verify without having to buy Mac or Windows licenses ... guess
> on every machine you could whip up a Ubuntu VM for verification. Just
> thinking about it ... perhaps it would be best to create a Docker image for
> doing the reproducible stuff ... 
just to be clear: the difference is on newline only, then Windows vs anything 
else. You get the same result on Linux, MacOS, FreeBSD, or any other Unix.

And if I want to be complete, if you get a source tarball from Windows, 
extract it to a Linux/MacOS/... box and build with "mvn -
Dline.separator='\r\n'", in general, you get the same result as a build under 
Windows: I tested multiple times, it worked, but we'll see if it works always 
or just "in general"

> Are there any plans on creating a plugin to allow verification?
> 
> Sort of something like this:
> "mvn package release:verify-reproduicble
> -DstagingRepoUrl=a.b.c.de/repo/blahblahblah"
> (Which doesn't deploy the
> artifacts, but instead download them and do a binary comparison) 
yes, but for now it was completely another form: this is the "Buildinfo file" 
proposal https://reproducible-builds.org/docs/jvm/
that I stopped to work on 1 year ago given I had no chance to get the same 
output: it's now a good time to restart working on it as next steps


> Also it could be great if the release-plugin could automatically set the
> property:
> a) if it finds the "project.build.outputTimestamp" set to some
> placeholder value b) if some switch tells it to prepare a reproducible
> build by using some sort of "switch" parameter 
> Guess that would sort of close the loop to get the biggest benefit out of
> the reproducible builds.
+1
issue has been created https://issues.apache.org/jira/browse/MRELEASE-1029
But I didn't work on it yet, help welcome

> I would be happy to help as I think this is one
> of the greatest new features. (Ok ... perhaps besides the
> sound-output-extension I learned about yesterday ;-) ) 
+1
the current step is important, but it's only the beginning of the story from 
an effective usage perspective

Regards,

Hervé

> 
> Chris
> 
> 
> Am 01.11.19, 09:24 schrieb "Hervé BOUTEMY" <he...@free.fr>:
> 
>     Le jeudi 31 octobre 2019, 17:26:52 CET Christofer Dutz a écrit :
> 
>     > Hi all,
>     > 
>     > as I can see you're voting on releasing the reproducible build
>     > extended
>     > plugin versions.
> 
>      Is there any documentation on how to use this new
> 
>     > feature?
>     > 
>     > I had a look at the confluence page, but that seemed like a
>     > brainstorming
>     > session.
> 
>     ok, the Wiki page [1] started as a brainstorming session, was updated to
> a proposal (the "Output Archive Entries Timestamp" parapgraph).
 And now I
> probably should order paragraph a little bit, and add a "Making your build
> reproducible" section for end uses to have a quick explanation. 
>     I'll write the explanation here as a first try before working on the
> Wiki:
 
>     1. upgrade your plugins to reproducible version, particularly
> mpaven-source-plugin, maven-jar-plugin and maven-assembly-plugin to vesion
> 3.2.0
 2. add project.build.outputTimestamp property with the timestamp
> value that will be used in zip/jar/tar archives: <properties>
>        
> <project.build.outputTimestamp>2019-10-02T08:04:00Z</project.build.outputTi
> mestamp>
 </properties>
>     
>     Notice:
>     - there is no Maven version prerequisite, everything happens at plugins
> level
 - Reproducible Builds require to have no version ranges in
> dependencies, generally gives different result on Unixes vs Windows, and
> generally depend on the major version of JDK used to compile (see "Out of
> Scope" paragraph) 
>     You have the basis configured, the output should be reproducible now.
>     If something is still not reproducible, use diffoscope to find the
> unstable output, find the plugin that generated this output and check if
> there is a reproducible version available: if not, please open an issue to
> help plugin maintainers improving Reproducible Builds support at every
> plugin level.
 
>     [1] 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318
 
>     
>      
> 
>     > I would love to add this to the PLC4X build asap.
> 
>     I'd love to have feedback
>     Don't hesitate to ping me.
> 
>     > 
>     > So I would like to test the release-candidates and vote too.
> 
>     I would love to have many tester and votes :)
>     
> 
>     > 
>     > Chris
>     > 
>     > 
>     > 
>     > Am 16.10.19, 14:42 schrieb "Hervé BOUTEMY" <he...@free.fr>:
>     > 
>     > 
>     >     Le mercredi 16 octobre 2019, 13:40:48 CEST Andreas Sewe a écrit :
>     > 
>     > 
>     > 
>     >     > Emmanuel Bourg wrote:
>     >     > 
>     >     > 
>     >     > > Le 16/10/2019 à 08:35, Hervé BOUTEMY a écrit :
>     >     > > 
>     >     > > 
>     >     > >> last question: now that we seem to fully understand each
>     >     > >> other,
>     >     > >> does it
>     >     > >> mean that you don't need any more "seconds since the epoch"
>     >     > >> format
>     >     > >> support for the property?
>     >     > > 
>     >     > > 
>     >     > > 
>     >     > > If Maven supports the SOURCE_DATE_EPOCH environment variable
>     >     > > I
>     >     > > don't
>     >     > > think that's necessary, otherwise it would be nice to be able
>     >     > > to
>     >     > > invoke
>     >     > > 
>     >     > > Maven with:
>     >     > > 
>     >     > > 
>     >     > >    mvn package
>     >     > >    -Dproject.build.outputTimestamp=$SOURCE_DATE_EPOCH
>     >     > > 
>     >     > > 
>     >     > > 
>     >     > > and this means supporting a timestamp formatted as seconds
>     >     > > since
>     >     > > the
>     >     > > epoch.
>     >     > 
>     >     > 
>     >     > 
>     >     > +1
>     >     > 
>     >     > The above would be a nice, simple way of bridging the gap
>     >     > between
>     >     > SOURCE_DATE_EPOCH and project.build.outputTimestamp.
>     > 
>     > 
>     > 
>     >     told like that, ok, why not
>     >     
>     > 
>     > 
>     > 
>     >     > 
>     >     > If it is not too much trouble to implement the "\d+ -> seconds
>     >     > since
>     >     > epoch" heuristic, them I would love to see it included.
>     > 
>     > 
>     > 
>     >     ok, I'll do and prepare the release
>     >     
>     >     Regards,
>     >     
>     >     Hervé
>     >     
>     > 
>     > 
>     > 
>     >     > 
>     >     > Best wishes,
>     >     > 
>     >     > Andreas
>     > 
>     > 
>     > 
>     >     
>     >     
>     >     
>     >     
>     >     
>     >     ------------------------------------------------------------------
>     >     ---
>     >     To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>     >     For additional commands, e-mail: dev-help@maven.apache.org
>     >     
>     >     
>     > 
>     > 
>     > 
>     > ---------------------------------------------------------------------
>     > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>     > For additional commands, e-mail: dev-help@maven.apache.org
>     > 
> 
>     
>     
>     
>     
>     
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>     For additional commands, e-mail: dev-help@maven.apache.org
>     
>     
> 





---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Re: last review of Reproducible Builds proposal

Posted by Hervé BOUTEMY <he...@free.fr>.
please provide an example of a visible issue so we can work on it

Regards,

Hervé

Le vendredi 1 novembre 2019, 10:27:36 CET Vladimir Sitnikov a écrit :
> OpenJDK8 uses hashmap for manifest entries, so jar files are not really
> reproducible there.
> 
> Note: there's run-to-run randomization in j.u.HashMap, so the manifest
> contents is not predictable.
> 
> Vladimir





---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Re: last review of Reproducible Builds proposal

Posted by Vladimir Sitnikov <si...@gmail.com>.
OpenJDK8 uses hashmap for manifest entries, so jar files are not really
reproducible there.

Note: there's run-to-run randomization in j.u.HashMap, so the manifest
contents is not predictable.

Vladimir

Re: last review of Reproducible Builds proposal

Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Herve,

thanks for that will try it asap and report any findings back.

But good to know that there is a difference between JDK major versions and OSes ... so it would probably be best to stage releases on Linux with an OpenJDK of the minimum supported version?
Just thinking how to make it possible to verify without having to buy Mac or Windows licenses ... guess on every machine you could whip up a Ubuntu VM for verification.
Just thinking about it ... perhaps it would be best to create a Docker image for doing the reproducible stuff ...

Are there any plans on creating a plugin to allow verification?

Sort of something like this:
"mvn package release:verify-reproduicble -DstagingRepoUrl=a.b.c.de/repo/blahblahblah"
(Which doesn't deploy the artifacts, but instead download them and do a binary comparison)

Also it could be great if the release-plugin could automatically set the property:
a) if it finds the "project.build.outputTimestamp" set to some placeholder value
b) if some switch tells it to prepare a reproducible build by using some sort of "switch" parameter

Guess that would sort of close the loop to get the biggest benefit out of the reproducible builds.
I would be happy to help as I think this is one of the greatest new features.
(Ok ... perhaps besides the sound-output-extension I learned about yesterday ;-) )


Chris


Am 01.11.19, 09:24 schrieb "Hervé BOUTEMY" <he...@free.fr>:

    Le jeudi 31 octobre 2019, 17:26:52 CET Christofer Dutz a écrit :
    > Hi all,
    > 
    > as I can see you're voting on releasing the reproducible build extended
    > plugin versions.
     Is there any documentation on how to use this new
    > feature?
    > 
    > I had a look at the confluence page, but that seemed like a brainstorming
    > session.
    ok, the Wiki page [1] started as a brainstorming session, was updated to a proposal (the "Output Archive Entries Timestamp" parapgraph).
    And now I probably should order paragraph a little bit, and add a "Making your build reproducible" section for end uses to have a quick explanation.
    
    I'll write the explanation here as a first try before working on the Wiki:
    
    1. upgrade your plugins to reproducible version, particularly mpaven-source-plugin, maven-jar-plugin and maven-assembly-plugin to vesion 3.2.0
    2. add project.build.outputTimestamp property with the timestamp value that will be used in zip/jar/tar archives:
      <properties>
        <project.build.outputTimestamp>2019-10-02T08:04:00Z</project.build.outputTimestamp>
      </properties>
    
    Notice:
    - there is no Maven version prerequisite, everything happens at plugins level
    - Reproducible Builds require to have no version ranges in dependencies, generally gives different result on Unixes vs Windows, and generally depend on the major version of JDK used to compile (see "Out of Scope" paragraph)
    
    You have the basis configured, the output should be reproducible now.
    If something is still not reproducible, use diffoscope to find the unstable output, find the plugin that generated this output and check if there is a reproducible version available: if not, please open an issue to help plugin maintainers improving Reproducible Builds support at every plugin level.
    
    [1]  https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318
    
    
     
    > I would love to add this to the PLC4X build asap.
    I'd love to have feedback
    Don't hesitate to ping me.
    > 
    > So I would like to test the release-candidates and vote too.
    I would love to have many tester and votes :)
    
    > 
    > Chris
    > 
    > 
    > 
    > Am 16.10.19, 14:42 schrieb "Hervé BOUTEMY" <he...@free.fr>:
    > 
    >     Le mercredi 16 octobre 2019, 13:40:48 CEST Andreas Sewe a écrit :
    > 
    >     > Emmanuel Bourg wrote:
    >     > 
    >     > > Le 16/10/2019 à 08:35, Hervé BOUTEMY a écrit :
    >     > > 
    >     > >> last question: now that we seem to fully understand each other,
    >     > >> does it
    >     > >> mean that you don't need any more "seconds since the epoch" format
    >     > >> support for the property?
    >     > > 
    >     > > 
    >     > > If Maven supports the SOURCE_DATE_EPOCH environment variable I
    >     > > don't
    >     > > think that's necessary, otherwise it would be nice to be able to
    >     > > invoke
    >     > > 
    >     > > Maven with:
    >     > > 
    >     > >    mvn package -Dproject.build.outputTimestamp=$SOURCE_DATE_EPOCH
    >     > > 
    >     > > 
    >     > > and this means supporting a timestamp formatted as seconds since
    >     > > the
    >     > > epoch.
    >     > 
    >     > 
    >     > +1
    >     > 
    >     > The above would be a nice, simple way of bridging the gap between
    >     > SOURCE_DATE_EPOCH and project.build.outputTimestamp.
    > 
    >     told like that, ok, why not
    >     
    > 
    >     > 
    >     > If it is not too much trouble to implement the "\d+ -> seconds since
    >     > epoch" heuristic, them I would love to see it included.
    > 
    >     ok, I'll do and prepare the release
    >     
    >     Regards,
    >     
    >     Hervé
    >     
    > 
    >     > 
    >     > Best wishes,
    >     > 
    >     > Andreas
    > 
    >     
    >     
    >     
    >     
    >     
    >     ---------------------------------------------------------------------
    >     To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
    >     For additional commands, e-mail: dev-help@maven.apache.org
    >     
    >     
    > 
    > 
    > ---------------------------------------------------------------------
    > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
    > For additional commands, e-mail: dev-help@maven.apache.org
    > 
    
    
    
    
    
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
    For additional commands, e-mail: dev-help@maven.apache.org