Documenting release testing?

[Stian Soiland-Reyes <st...@apache.org>]

Perhaps we should try to write it up and (propose an) update for
http://incubator.apache.org/guides/releasemanagement.html -- there
would be many differences of opinions!

BTW, all incubator committers can contribute to improve the Incubator
site as well using https://cms.apache.org/incubator/ - just be careful
about policy documents.


On 1 March 2016 at 22:42, Gale Naylor <Ga...@noventussolutions.com> wrote:
> Thank you, Stian! Some of my questions I figured out today, but some I did
> not, so I very much appreciate the hints and instructions.
>
>
> On Tue, Mar 1, 2016 at 2:28 PM Stian Soiland-Reyes <st...@apache.org> wrote:
>
>> Thanks for reviewing!
>>
>>
>>  > (1) How do I verify that the commit id in the downloaded files matches
>> that
>> > in the VOTE email? (I've looked on the internet, but have yet to find
>> > anything helpful.)
>>
>> I don't think most people check this deeply.. but I guess at least one
>> voter should.
>>
>> Here's what I do:
>>
>> mkdir 1 ; cd 1  # new folder
>> git clone that-repository
>> git checkout that-commit-id-from-the-email-asdfjaskdjfsakjdfksajdf
>> rm -rf *
>> unzip ../the-release-candidate.zip
>> mv apache-taverna-*/* .  (one level up)
>> git status
>>
>> Git will then check the checksums of every file and let you know what
>> has 'changed' (as it would believe you have edited it).
>>
>>
>> Here's another way that doesn't require using the 'git' command:
>>
>> Download the git commit corresponding to the email by browsing for it on
>> GitHub:
>>
>>
>> https://github.com/apache/incubator-taverna-language/tree/66866a5454ed23262c055f65155d7a195c68a17d
>> Click "Download ZIP"
>>
>> mkdir 1 ; cd 1
>> unzip ../66866a5454ed23262c055f65155d7a195c68a17d.zip
>>
>> cd ../ ; mkdir 2 ; cd 2
>> unzip release-candidate.zip
>>
>> cd ..
>> diff -uR 1 2
>>
>> The files that differ (and their differences!) will be shown.
>>
>> Make sure you don't have any target/ folders before diff-ing (run mvn
>> clean to be sure)
>>
>> If you do the above with a git clone instead - remember that the zip
>> doesn't include the .git/ folder - so you would have to delete the
>> checked out .git folder before diffing.  (Don't do this on your
>> regular checkout as you would lose all local branches!)
>>
>>
>>
>> > (2) Are the "binary artifacts" in the target folders? Which files are
>> > considered "binary artifacts?"
>>
>> Well, the target/ files are binary artifacts, but they (should) have
>> been made by your build on your machine - not be part of the source
>> ZIP.
>>
>>
>> One thing to look out for is in the downloaded source ZIP that there
>> are no unexpected binary artifacts in it *before you build* - e.g.
>> there should not be any *.jars in there.  (The source distribution
>> should be 'clean'). We do have some *expected* binaries, pictures and
>> test workflows for instance.  As those can't have license headers they
>> should be declared in NOTICE/LICENSE if they came from third-parties.
>> (E.g. if we used a Creative Commons-licensed JPEG picture)
>>
>>
>> In terms of release candidate, the binaries would be installers and
>> JARs etc., under
>> https://dist.apache.org/repos/dist/dev/incubator/taverna/binaries/
>> (But there are none for this release candidate)
>>
>> ..in addition to the JARs that have been staged to the Maven repository
>>
>> https://repository.apache.org/content/repositories/orgapachetaverna-1011/org/apache/taverna/
>>
>>
>> > (3) How do I verify that the build produces the binaries? By visual
>> > inspection? What am I looking for?
>>
>>
>> As for checking the Maven repository, if you want to do it really
>> proper you should check that all the JARs that are staged can be built
>> from the downloaded release candidate ZIP - e.g. that your target/
>> folder contains all of the same ones.   If I do this, I do a recursive
>> wget of the repository, and then compare the result of "find . -name
>> '*jar'"  in the wget-tree with */*/target/*.jar.  I don't think most
>> people do this.
>>
>> Paranoid-mode would be to download each JAR and check that they only
>> have the same *.class files - but these would differ for every build
>> and so can't be compared any further without lots of clever tooling -
>> so nobody does this. (I think there should be an Apache-hosted tool or
>> Maven plugin that could do this).
>>
>>
>> Practically the best is just to click briefly into the repository in a
>> browser and see there are not any 'additional' folders that shouldn't
>> be there, e.g. we are now voting on taverna-maven-parent, taverna-osgi
>> and taverna-language, and so we should not see
>> org/apache/taverna/engine in there - as that is a group Id from
>> taverna-engine.
>>
>> (We have already changed the groupIDs to correspond to the repository
>> which corresponds to the release name, so at least that correspondance
>> is easy to check on Taverna, but not so on many other projects).
>>
>>
>> As binary releases from Apache Software Foundation are considered
>> "convenience only" they are not crucial for the vote - the source
>> release is the golden thing which everything else should be made from.
>> Practically speaking "everyone" uses the JARs from Maven repository
>> though, so I wouldn't dismiss them totally - at least one person in
>> the vote should do such a check.
>>
>>
>> > (4) How do I check the dependencies?
>>
>> mvn dependency:tree gives a nice list - but what should you check for?
>> Well, it's mainly about licensing -
>> http://www.apache.org/legal/resolved.html lists what is compatible as
>> dependencies of an ASF release.  Now you don't need to go through the
>> list - but sometimes there are Well Known forbidden dependencies that
>> People (tm) recognize -e.g. mysql-connector and Hibernate are banned
>> as they are (L)GPL.
>>
>> Luckily there's another Maven plugin that can do the job:
>>
>> mvn license:aggregate-add-third-party
>>
>> cat target/generated-sources/license/THIRD-PARTY.txt | sort
>>
>>      (Aduna BSD license) OpenRDF Sesame: HTTP client
>> (org.openrdf.sesame:sesame-http-client:2.7.0 -
>> http://www.openrdf.org/sesame-core/sesame-http/sesame-http-client/)
>>      (Aduna BSD license) OpenRDF Sesame: HTTP protocol
>> (org.openrdf.sesame:sesame-http-protocol:2.7.0 -
>> http://www.openrdf.org/sesame-core/sesame-http/sesame-http-protocol/)
>> (..)
>>      (The Apache Software License, Version 2.0) Xerces2-j
>> (xerces:xercesImpl:2.11.0 - https://xerces.apache.org/xerces2-j/)
>>      (Unknown license) commons-beanutils
>> (commons-beanutils:commons-beanutils:1.7.0 - no url defined)
>>      (Unknown license) com.springsource.org.jaxen
>> (org.jaxen:com.springsource.org.jaxen:1.1.1 - no url defined)
>>      (Unknown license) com.springsource.org.jdom
>> (org.jdom:com.springsource.org.jdom:1.1.0 - no url defined)
>>      (Unknown license) Logging (commons-logging:commons-logging:1.0.3
>> - http://jakarta.apache.org/commons/logging/)
>>
>> (BTW, those last 4 are already checked to be OK, see
>> http://dev.mygrid.org.uk/wiki/display/developer/Third-party+licenses )
>>
>>
>> > Regarding the build output: Since this is the first time I've done this,
>> I
>> > don't know what's okay to ignore. Here is a summary of the warning
>> messages
>> > I received when I ran mvn clean install. I sent the output to two
>> different
>> > files using the following command (Windows 10/ GitBash): mvn clean
>> install
>> >> output1.txt 2> output2.txt. I appreciate any insight.
>>
>> Great!   I think those should be tracked in JIRA as we want to reduce
>> warnings.
>>
>>
>>
>> Generally with Maven, if it finishes with a big SUCCESS, then that's
>> true. The warnings are more like warnings for the developers doing
>> bad-practice-stuff than warnings about something going wrong with the
>> build.   Often the fixes are simple, like adding a @Deprecated tag
>> where you delibately use old APIs, or actually follow the fix
>> suggested by the warning.
>>
>> I think we want to follow Andy's advice and "release early, release
>> often" - which entails a "good enough" - not "super-perfect".
>> Obviously each committer votes independenly by their own quality
>> measures.
>>
>> While Apache Software Foundation always says that community is king -
>> the Apache name is still recognized by the public as a kind of
>> "quality mark" - if that is deserved or not I won't comment on, but of
>> course there is also a sense of pride in that we don't want to set the
>> standard too low.  :)
>>
>> (E.g. Taverna just cancelled 3 release candidates as they didn't pass
>> all their tests on Windows - but the community of another Apache
>> project might not consider Windows important enough to halt a release)
>>
>>
>>
>>
>> --
>> Stian Soiland-Reyes
>> Apache Taverna (incubating), Apache Commons RDF (incubating)
>> http://orcid.org/0000-0001-9842-9718
>>



-- 
Stian Soiland-Reyes
Apache Taverna (incubating), Apache Commons RDF (incubating)
http://orcid.org/0000-0001-9842-9718

Re: Documenting release testing?

[Gale Naylor <Ga...@noventussolutions.com>]

I'm planning on writing something up. It may take a couple of drafts. ;-)
Thanks, I didn't know committers could contribute to the incubator site.

On Tue, Mar 1, 2016 at 3:26 PM Stian Soiland-Reyes <st...@apache.org> wrote:

> Perhaps we should try to write it up and (propose an) update for
> http://incubator.apache.org/guides/releasemanagement.html -- there
> would be many differences of opinions!
>
> BTW, all incubator committers can contribute to improve the Incubator
> site as well using https://cms.apache.org/incubator/ - just be careful
> about policy documents.
>
>
> On 1 March 2016 at 22:42, Gale Naylor <Ga...@noventussolutions.com> wrote:
> > Thank you, Stian! Some of my questions I figured out today, but some I
> did
> > not, so I very much appreciate the hints and instructions.
> >
> >
> > On Tue, Mar 1, 2016 at 2:28 PM Stian Soiland-Reyes <st...@apache.org>
> wrote:
> >
> >> Thanks for reviewing!
> >>
> >>
> >>  > (1) How do I verify that the commit id in the downloaded files
> matches
> >> that
> >> > in the VOTE email? (I've looked on the internet, but have yet to find
> >> > anything helpful.)
> >>
> >> I don't think most people check this deeply.. but I guess at least one
> >> voter should.
> >>
> >> Here's what I do:
> >>
> >> mkdir 1 ; cd 1  # new folder
> >> git clone that-repository
> >> git checkout that-commit-id-from-the-email-asdfjaskdjfsakjdfksajdf
> >> rm -rf *
> >> unzip ../the-release-candidate.zip
> >> mv apache-taverna-*/* .  (one level up)
> >> git status
> >>
> >> Git will then check the checksums of every file and let you know what
> >> has 'changed' (as it would believe you have edited it).
> >>
> >>
> >> Here's another way that doesn't require using the 'git' command:
> >>
> >> Download the git commit corresponding to the email by browsing for it on
> >> GitHub:
> >>
> >>
> >>
> https://github.com/apache/incubator-taverna-language/tree/66866a5454ed23262c055f65155d7a195c68a17d
> >> Click "Download ZIP"
> >>
> >> mkdir 1 ; cd 1
> >> unzip ../66866a5454ed23262c055f65155d7a195c68a17d.zip
> >>
> >> cd ../ ; mkdir 2 ; cd 2
> >> unzip release-candidate.zip
> >>
> >> cd ..
> >> diff -uR 1 2
> >>
> >> The files that differ (and their differences!) will be shown.
> >>
> >> Make sure you don't have any target/ folders before diff-ing (run mvn
> >> clean to be sure)
> >>
> >> If you do the above with a git clone instead - remember that the zip
> >> doesn't include the .git/ folder - so you would have to delete the
> >> checked out .git folder before diffing.  (Don't do this on your
> >> regular checkout as you would lose all local branches!)
> >>
> >>
> >>
> >> > (2) Are the "binary artifacts" in the target folders? Which files are
> >> > considered "binary artifacts?"
> >>
> >> Well, the target/ files are binary artifacts, but they (should) have
> >> been made by your build on your machine - not be part of the source
> >> ZIP.
> >>
> >>
> >> One thing to look out for is in the downloaded source ZIP that there
> >> are no unexpected binary artifacts in it *before you build* - e.g.
> >> there should not be any *.jars in there.  (The source distribution
> >> should be 'clean'). We do have some *expected* binaries, pictures and
> >> test workflows for instance.  As those can't have license headers they
> >> should be declared in NOTICE/LICENSE if they came from third-parties.
> >> (E.g. if we used a Creative Commons-licensed JPEG picture)
> >>
> >>
> >> In terms of release candidate, the binaries would be installers and
> >> JARs etc., under
> >> https://dist.apache.org/repos/dist/dev/incubator/taverna/binaries/
> >> (But there are none for this release candidate)
> >>
> >> ..in addition to the JARs that have been staged to the Maven repository
> >>
> >>
> https://repository.apache.org/content/repositories/orgapachetaverna-1011/org/apache/taverna/
> >>
> >>
> >> > (3) How do I verify that the build produces the binaries? By visual
> >> > inspection? What am I looking for?
> >>
> >>
> >> As for checking the Maven repository, if you want to do it really
> >> proper you should check that all the JARs that are staged can be built
> >> from the downloaded release candidate ZIP - e.g. that your target/
> >> folder contains all of the same ones.   If I do this, I do a recursive
> >> wget of the repository, and then compare the result of "find . -name
> >> '*jar'"  in the wget-tree with */*/target/*.jar.  I don't think most
> >> people do this.
> >>
> >> Paranoid-mode would be to download each JAR and check that they only
> >> have the same *.class files - but these would differ for every build
> >> and so can't be compared any further without lots of clever tooling -
> >> so nobody does this. (I think there should be an Apache-hosted tool or
> >> Maven plugin that could do this).
> >>
> >>
> >> Practically the best is just to click briefly into the repository in a
> >> browser and see there are not any 'additional' folders that shouldn't
> >> be there, e.g. we are now voting on taverna-maven-parent, taverna-osgi
> >> and taverna-language, and so we should not see
> >> org/apache/taverna/engine in there - as that is a group Id from
> >> taverna-engine.
> >>
> >> (We have already changed the groupIDs to correspond to the repository
> >> which corresponds to the release name, so at least that correspondance
> >> is easy to check on Taverna, but not so on many other projects).
> >>
> >>
> >> As binary releases from Apache Software Foundation are considered
> >> "convenience only" they are not crucial for the vote - the source
> >> release is the golden thing which everything else should be made from.
> >> Practically speaking "everyone" uses the JARs from Maven repository
> >> though, so I wouldn't dismiss them totally - at least one person in
> >> the vote should do such a check.
> >>
> >>
> >> > (4) How do I check the dependencies?
> >>
> >> mvn dependency:tree gives a nice list - but what should you check for?
> >> Well, it's mainly about licensing -
> >> http://www.apache.org/legal/resolved.html lists what is compatible as
> >> dependencies of an ASF release.  Now you don't need to go through the
> >> list - but sometimes there are Well Known forbidden dependencies that
> >> People (tm) recognize -e.g. mysql-connector and Hibernate are banned
> >> as they are (L)GPL.
> >>
> >> Luckily there's another Maven plugin that can do the job:
> >>
> >> mvn license:aggregate-add-third-party
> >>
> >> cat target/generated-sources/license/THIRD-PARTY.txt | sort
> >>
> >>      (Aduna BSD license) OpenRDF Sesame: HTTP client
> >> (org.openrdf.sesame:sesame-http-client:2.7.0 -
> >> http://www.openrdf.org/sesame-core/sesame-http/sesame-http-client/)
> >>      (Aduna BSD license) OpenRDF Sesame: HTTP protocol
> >> (org.openrdf.sesame:sesame-http-protocol:2.7.0 -
> >> http://www.openrdf.org/sesame-core/sesame-http/sesame-http-protocol/)
> >> (..)
> >>      (The Apache Software License, Version 2.0) Xerces2-j
> >> (xerces:xercesImpl:2.11.0 - https://xerces.apache.org/xerces2-j/)
> >>      (Unknown license) commons-beanutils
> >> (commons-beanutils:commons-beanutils:1.7.0 - no url defined)
> >>      (Unknown license) com.springsource.org.jaxen
> >> (org.jaxen:com.springsource.org.jaxen:1.1.1 - no url defined)
> >>      (Unknown license) com.springsource.org.jdom
> >> (org.jdom:com.springsource.org.jdom:1.1.0 - no url defined)
> >>      (Unknown license) Logging (commons-logging:commons-logging:1.0.3
> >> - http://jakarta.apache.org/commons/logging/)
> >>
> >> (BTW, those last 4 are already checked to be OK, see
> >> http://dev.mygrid.org.uk/wiki/display/developer/Third-party+licenses )
> >>
> >>
> >> > Regarding the build output: Since this is the first time I've done
> this,
> >> I
> >> > don't know what's okay to ignore. Here is a summary of the warning
> >> messages
> >> > I received when I ran mvn clean install. I sent the output to two
> >> different
> >> > files using the following command (Windows 10/ GitBash): mvn clean
> >> install
> >> >> output1.txt 2> output2.txt. I appreciate any insight.
> >>
> >> Great!   I think those should be tracked in JIRA as we want to reduce
> >> warnings.
> >>
> >>
> >>
> >> Generally with Maven, if it finishes with a big SUCCESS, then that's
> >> true. The warnings are more like warnings for the developers doing
> >> bad-practice-stuff than warnings about something going wrong with the
> >> build.   Often the fixes are simple, like adding a @Deprecated tag
> >> where you delibately use old APIs, or actually follow the fix
> >> suggested by the warning.
> >>
> >> I think we want to follow Andy's advice and "release early, release
> >> often" - which entails a "good enough" - not "super-perfect".
> >> Obviously each committer votes independenly by their own quality
> >> measures.
> >>
> >> While Apache Software Foundation always says that community is king -
> >> the Apache name is still recognized by the public as a kind of
> >> "quality mark" - if that is deserved or not I won't comment on, but of
> >> course there is also a sense of pride in that we don't want to set the
> >> standard too low.  :)
> >>
> >> (E.g. Taverna just cancelled 3 release candidates as they didn't pass
> >> all their tests on Windows - but the community of another Apache
> >> project might not consider Windows important enough to halt a release)
> >>
> >>
> >>
> >>
> >> --
> >> Stian Soiland-Reyes
> >> Apache Taverna (incubating), Apache Commons RDF (incubating)
> >> http://orcid.org/0000-0001-9842-9718
> >>
>
>
>
> --
> Stian Soiland-Reyes
> Apache Taverna (incubating), Apache Commons RDF (incubating)
> http://orcid.org/0000-0001-9842-9718
>