pack200 "normalize" only shrinks size by about 1 %

[Marshall Schor <ms...@schor.com>]

I tried it on a bunch of Jars, including the big "runtime" plugin jar.

It shrunk the result by about 1 % vs not running pack200 -repack.

Because it's so little, I think it's not worth doing.

-Marshall



On 9/20/2017 11:34 AM, Marshall Schor wrote:
> I'm starting to think about redoing our Eclipse Plugin release flow.  I'm hoping
> to start a thread where people can post answers to issues (there are many...).
>
> Motivation: start using the Symantec signing process for Jars (in addition to
> the PGP Apache signing process), so that Eclipse, when installing new plugins
> doesn't warn about unsigned artifacts.
>
> There are many things to figure out.  One is how to manage the cost of the
> signing. Signing costs real money, which Apache covers (see
> https://reference.apache.org/pmc/codesigning). 
>
> 1) To find out:  is the cost per Jar, or per "signing event" which can include
> multiple Jars?  This can affect some design choices.
>
> 2) Another thing is when (in our release flow) to sign.  The signing process
> alters the JARs.
>
> Our current flow is  1) generate candidate with PGP signature, 2) Vote  3)
> repeat if failed rc, otherwise promote.  An Apache principle is to have the Vote
> be on the signed artifacts, which are then promoted without modification.  This
> implies the step 1) would need to include signing. 
>
> This means that every failed release candidate costs money.  I found out that
> some projects "accept" this, but have low failed percentages, like 1 in 10.  In
> our process, we seem to frequently have multiple release candidates per release. 
>
> Perhaps a way forward is to have a process for the Eclipse plugin release which
> is different, and isn't invoked until a "preliminary" candidate review has been
> done; once this has happened, and there's confidence that the signed version
> would likely pass the vote, an official Eclipse plugin release candidate could
> be posted with signed JARs.
>
> 3) Another thing to figure out, assuming that signing costs "per jar", is to
> insure only new Jars are signed, not the older versions.  This could affect
> efforts to limit the number of JARs (or not).
>
> 4) Another thing to figure out is what needs to be signed.  I think that only
> Eclipse plugin jars need signing, and not the other jars associated with the
> update site, like artifacts.jar, content.jar, and for each feature,
> feat-name.jar (e.g. org.apache.uima.runtime_3.0.0.alpha02.jar).  But maybe
> that's wrong.  The site https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F
> says multiple things: 1) "by default, every Jar pushed to an update site is
> signed", 2) "in standalone zip distributions, all JARed plugins will be
> signed".  So, my guess is that you don't need to sign the other kinds of JARs. 
>
> Some websites imply that JARs other than the plugin ones are signed (
> https://wiki.eclipse.org/Platform-releng-signedbuild ), but it's unclear if they
> need signing in order to get the Eclipse install program not to complain.  That
> website says the master zip having all the plugins and features is sent to the
> signing process.
>
> 5) Some of our Jars (typically, the runtime Jars) have other Jars inside them. 
> This site https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F says "JARs
> nested at arbitrary depth within other JARs" are signed, too.  Is that needed to
> make the Eclipse install program not complain about installing unsigned artifacts?
>
> 6) One site ( https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F ) says
> there may be legal reasons you are not permitted to sign an embedded JAR.  I've
> never seen anything about that, and don't know how to determine if such a
> prohibition exists.  This page says, under what gets signed, "Some JARs may be
> excluded if there are technical or legal reasons why they cannot be signed".
>
> 7) Signing and pack200 interact.  This page
> https://wiki.eclipse.org/Tycho/Pack200#Pack200_and_Signing describes a flow:
> 1. pack200 operated to "normalize" - this is specified by the "repack" option
> 2. sign
> 3. pack200 operated to generate the compressed version
> 4. p2 metadata generation / updating
>
> Our current eclipse plugin build flow runs pack200 with both the "-repack" and
> "-pack" options.  I think this would need to be split into two steps (assuming
> we keep pack200 - see following), with the results from -repack being sent for
> signing, and then the result subsequently packed.
>
> 8) I question if we need to do pack200 packing.  It has these effects:
> 1. for every Jar, it generates 2 Jars: the original and the packed.  So there
> are 2 jars that need "signing", thus it costs twice as much. 
> 2. A typical Jar (caseditor.ide) is 69K bytes, packed 28K.  A big jar
> uima-runtime is 4M, packed: 1.3M
> So the result is increased space on the distribution medium, but decreased
> bandwidth for downloading.  But the download frequency is very low - only when
> someone installs the UIMA Eclipse plugins into a new Eclipse instance.  I think
> pack200 was envisioned to help websites where the fetch was more frequent, and
> web-page load times were important. 
>
> Should we abandon doing pack200 packaging?
>
> Well, those 8 topics should get us started.  Perhaps it will be best to start a
> new thread for each one.
> I'm looking forward to the discussion. 
>
> Cheers. -Marshall
>