redesigining our Eclipse plugin release build flow - list of issues to find out about

[Marshall Schor <ms...@schor.com>]

I'm starting to think about redoing our Eclipse Plugin release flow.  I'm hoping
to start a thread where people can post answers to issues (there are many...).

Motivation: start using the Symantec signing process for Jars (in addition to
the PGP Apache signing process), so that Eclipse, when installing new plugins
doesn't warn about unsigned artifacts.

There are many things to figure out.  One is how to manage the cost of the
signing. Signing costs real money, which Apache covers (see
https://reference.apache.org/pmc/codesigning). 

1) To find out:  is the cost per Jar, or per "signing event" which can include
multiple Jars?  This can affect some design choices.

2) Another thing is when (in our release flow) to sign.  The signing process
alters the JARs.

Our current flow is  1) generate candidate with PGP signature, 2) Vote  3)
repeat if failed rc, otherwise promote.  An Apache principle is to have the Vote
be on the signed artifacts, which are then promoted without modification.  This
implies the step 1) would need to include signing. 

This means that every failed release candidate costs money.  I found out that
some projects "accept" this, but have low failed percentages, like 1 in 10.  In
our process, we seem to frequently have multiple release candidates per release. 

Perhaps a way forward is to have a process for the Eclipse plugin release which
is different, and isn't invoked until a "preliminary" candidate review has been
done; once this has happened, and there's confidence that the signed version
would likely pass the vote, an official Eclipse plugin release candidate could
be posted with signed JARs.

3) Another thing to figure out, assuming that signing costs "per jar", is to
insure only new Jars are signed, not the older versions.  This could affect
efforts to limit the number of JARs (or not).

4) Another thing to figure out is what needs to be signed.  I think that only
Eclipse plugin jars need signing, and not the other jars associated with the
update site, like artifacts.jar, content.jar, and for each feature,
feat-name.jar (e.g. org.apache.uima.runtime_3.0.0.alpha02.jar).  But maybe
that's wrong.  The site https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F
says multiple things: 1) "by default, every Jar pushed to an update site is
signed", 2) "in standalone zip distributions, all JARed plugins will be
signed".  So, my guess is that you don't need to sign the other kinds of JARs. 

Some websites imply that JARs other than the plugin ones are signed (
https://wiki.eclipse.org/Platform-releng-signedbuild ), but it's unclear if they
need signing in order to get the Eclipse install program not to complain.  That
website says the master zip having all the plugins and features is sent to the
signing process.

5) Some of our Jars (typically, the runtime Jars) have other Jars inside them. 
This site https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F says "JARs
nested at arbitrary depth within other JARs" are signed, too.  Is that needed to
make the Eclipse install program not complain about installing unsigned artifacts?

6) One site ( https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F ) says
there may be legal reasons you are not permitted to sign an embedded JAR.  I've
never seen anything about that, and don't know how to determine if such a
prohibition exists.  This page says, under what gets signed, "Some JARs may be
excluded if there are technical or legal reasons why they cannot be signed".

7) Signing and pack200 interact.  This page
https://wiki.eclipse.org/Tycho/Pack200#Pack200_and_Signing describes a flow:
1. pack200 operated to "normalize" - this is specified by the "repack" option
2. sign
3. pack200 operated to generate the compressed version
4. p2 metadata generation / updating

Our current eclipse plugin build flow runs pack200 with both the "-repack" and
"-pack" options.  I think this would need to be split into two steps (assuming
we keep pack200 - see following), with the results from -repack being sent for
signing, and then the result subsequently packed.

8) I question if we need to do pack200 packing.  It has these effects:
1. for every Jar, it generates 2 Jars: the original and the packed.  So there
are 2 jars that need "signing", thus it costs twice as much. 
2. A typical Jar (caseditor.ide) is 69K bytes, packed 28K.  A big jar
uima-runtime is 4M, packed: 1.3M
So the result is increased space on the distribution medium, but decreased
bandwidth for downloading.  But the download frequency is very low - only when
someone installs the UIMA Eclipse plugins into a new Eclipse instance.  I think
pack200 was envisioned to help websites where the fetch was more frequent, and
web-page load times were important. 

Should we abandon doing pack200 packaging?

Well, those 8 topics should get us started.  Perhaps it will be best to start a
new thread for each one.
I'm looking forward to the discussion. 

Cheers. -Marshall

pack200 "normalize" only shrinks size by about 1 %

[Marshall Schor <ms...@schor.com>]

I tried it on a bunch of Jars, including the big "runtime" plugin jar.

It shrunk the result by about 1 % vs not running pack200 -repack.

Because it's so little, I think it's not worth doing.

-Marshall



On 9/20/2017 11:34 AM, Marshall Schor wrote:
> I'm starting to think about redoing our Eclipse Plugin release flow.  I'm hoping
> to start a thread where people can post answers to issues (there are many...).
>
> Motivation: start using the Symantec signing process for Jars (in addition to
> the PGP Apache signing process), so that Eclipse, when installing new plugins
> doesn't warn about unsigned artifacts.
>
> There are many things to figure out.  One is how to manage the cost of the
> signing. Signing costs real money, which Apache covers (see
> https://reference.apache.org/pmc/codesigning). 
>
> 1) To find out:  is the cost per Jar, or per "signing event" which can include
> multiple Jars?  This can affect some design choices.
>
> 2) Another thing is when (in our release flow) to sign.  The signing process
> alters the JARs.
>
> Our current flow is  1) generate candidate with PGP signature, 2) Vote  3)
> repeat if failed rc, otherwise promote.  An Apache principle is to have the Vote
> be on the signed artifacts, which are then promoted without modification.  This
> implies the step 1) would need to include signing. 
>
> This means that every failed release candidate costs money.  I found out that
> some projects "accept" this, but have low failed percentages, like 1 in 10.  In
> our process, we seem to frequently have multiple release candidates per release. 
>
> Perhaps a way forward is to have a process for the Eclipse plugin release which
> is different, and isn't invoked until a "preliminary" candidate review has been
> done; once this has happened, and there's confidence that the signed version
> would likely pass the vote, an official Eclipse plugin release candidate could
> be posted with signed JARs.
>
> 3) Another thing to figure out, assuming that signing costs "per jar", is to
> insure only new Jars are signed, not the older versions.  This could affect
> efforts to limit the number of JARs (or not).
>
> 4) Another thing to figure out is what needs to be signed.  I think that only
> Eclipse plugin jars need signing, and not the other jars associated with the
> update site, like artifacts.jar, content.jar, and for each feature,
> feat-name.jar (e.g. org.apache.uima.runtime_3.0.0.alpha02.jar).  But maybe
> that's wrong.  The site https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F
> says multiple things: 1) "by default, every Jar pushed to an update site is
> signed", 2) "in standalone zip distributions, all JARed plugins will be
> signed".  So, my guess is that you don't need to sign the other kinds of JARs. 
>
> Some websites imply that JARs other than the plugin ones are signed (
> https://wiki.eclipse.org/Platform-releng-signedbuild ), but it's unclear if they
> need signing in order to get the Eclipse install program not to complain.  That
> website says the master zip having all the plugins and features is sent to the
> signing process.
>
> 5) Some of our Jars (typically, the runtime Jars) have other Jars inside them. 
> This site https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F says "JARs
> nested at arbitrary depth within other JARs" are signed, too.  Is that needed to
> make the Eclipse install program not complain about installing unsigned artifacts?
>
> 6) One site ( https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F ) says
> there may be legal reasons you are not permitted to sign an embedded JAR.  I've
> never seen anything about that, and don't know how to determine if such a
> prohibition exists.  This page says, under what gets signed, "Some JARs may be
> excluded if there are technical or legal reasons why they cannot be signed".
>
> 7) Signing and pack200 interact.  This page
> https://wiki.eclipse.org/Tycho/Pack200#Pack200_and_Signing describes a flow:
> 1. pack200 operated to "normalize" - this is specified by the "repack" option
> 2. sign
> 3. pack200 operated to generate the compressed version
> 4. p2 metadata generation / updating
>
> Our current eclipse plugin build flow runs pack200 with both the "-repack" and
> "-pack" options.  I think this would need to be split into two steps (assuming
> we keep pack200 - see following), with the results from -repack being sent for
> signing, and then the result subsequently packed.
>
> 8) I question if we need to do pack200 packing.  It has these effects:
> 1. for every Jar, it generates 2 Jars: the original and the packed.  So there
> are 2 jars that need "signing", thus it costs twice as much. 
> 2. A typical Jar (caseditor.ide) is 69K bytes, packed 28K.  A big jar
> uima-runtime is 4M, packed: 1.3M
> So the result is increased space on the distribution medium, but decreased
> bandwidth for downloading.  But the download frequency is very low - only when
> someone installs the UIMA Eclipse plugins into a new Eclipse instance.  I think
> pack200 was envisioned to help websites where the fetch was more frequent, and
> web-page load times were important. 
>
> Should we abandon doing pack200 packaging?
>
> Well, those 8 topics should get us started.  Perhaps it will be best to start a
> new thread for each one.
> I'm looking forward to the discussion. 
>
> Cheers. -Marshall
>

Eclipse plugin release signing - charge is per "signing event" which may include multiple JARs

[Marshall Schor <ms...@schor.com>]

Found an infra mail thread on this topic that says the charge is per "signing
event", and that can be for multiple JARs.

Infra email data 11/10/2016, author Mark Thomas and Robert Munteanu.

-Marshall



On 9/20/2017 11:34 AM, Marshall Schor wrote:
> I'm starting to think about redoing our Eclipse Plugin release flow.  I'm hoping
> to start a thread where people can post answers to issues (there are many...).
>
> Motivation: start using the Symantec signing process for Jars (in addition to
> the PGP Apache signing process), so that Eclipse, when installing new plugins
> doesn't warn about unsigned artifacts.
>
> There are many things to figure out.  One is how to manage the cost of the
> signing. Signing costs real money, which Apache covers (see
> https://reference.apache.org/pmc/codesigning). 
>
> 1) To find out:  is the cost per Jar, or per "signing event" which can include
> multiple Jars?  This can affect some design choices.
>
> 2) Another thing is when (in our release flow) to sign.  The signing process
> alters the JARs.
>
> Our current flow is  1) generate candidate with PGP signature, 2) Vote  3)
> repeat if failed rc, otherwise promote.  An Apache principle is to have the Vote
> be on the signed artifacts, which are then promoted without modification.  This
> implies the step 1) would need to include signing. 
>
> This means that every failed release candidate costs money.  I found out that
> some projects "accept" this, but have low failed percentages, like 1 in 10.  In
> our process, we seem to frequently have multiple release candidates per release. 
>
> Perhaps a way forward is to have a process for the Eclipse plugin release which
> is different, and isn't invoked until a "preliminary" candidate review has been
> done; once this has happened, and there's confidence that the signed version
> would likely pass the vote, an official Eclipse plugin release candidate could
> be posted with signed JARs.
>
> 3) Another thing to figure out, assuming that signing costs "per jar", is to
> insure only new Jars are signed, not the older versions.  This could affect
> efforts to limit the number of JARs (or not).
>
> 4) Another thing to figure out is what needs to be signed.  I think that only
> Eclipse plugin jars need signing, and not the other jars associated with the
> update site, like artifacts.jar, content.jar, and for each feature,
> feat-name.jar (e.g. org.apache.uima.runtime_3.0.0.alpha02.jar).  But maybe
> that's wrong.  The site https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F
> says multiple things: 1) "by default, every Jar pushed to an update site is
> signed", 2) "in standalone zip distributions, all JARed plugins will be
> signed".  So, my guess is that you don't need to sign the other kinds of JARs. 
>
> Some websites imply that JARs other than the plugin ones are signed (
> https://wiki.eclipse.org/Platform-releng-signedbuild ), but it's unclear if they
> need signing in order to get the Eclipse install program not to complain.  That
> website says the master zip having all the plugins and features is sent to the
> signing process.
>
> 5) Some of our Jars (typically, the runtime Jars) have other Jars inside them. 
> This site https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F says "JARs
> nested at arbitrary depth within other JARs" are signed, too.  Is that needed to
> make the Eclipse install program not complain about installing unsigned artifacts?
>
> 6) One site ( https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F ) says
> there may be legal reasons you are not permitted to sign an embedded JAR.  I've
> never seen anything about that, and don't know how to determine if such a
> prohibition exists.  This page says, under what gets signed, "Some JARs may be
> excluded if there are technical or legal reasons why they cannot be signed".
>
> 7) Signing and pack200 interact.  This page
> https://wiki.eclipse.org/Tycho/Pack200#Pack200_and_Signing describes a flow:
> 1. pack200 operated to "normalize" - this is specified by the "repack" option
> 2. sign
> 3. pack200 operated to generate the compressed version
> 4. p2 metadata generation / updating
>
> Our current eclipse plugin build flow runs pack200 with both the "-repack" and
> "-pack" options.  I think this would need to be split into two steps (assuming
> we keep pack200 - see following), with the results from -repack being sent for
> signing, and then the result subsequently packed.
>
> 8) I question if we need to do pack200 packing.  It has these effects:
> 1. for every Jar, it generates 2 Jars: the original and the packed.  So there
> are 2 jars that need "signing", thus it costs twice as much. 
> 2. A typical Jar (caseditor.ide) is 69K bytes, packed 28K.  A big jar
> uima-runtime is 4M, packed: 1.3M
> So the result is increased space on the distribution medium, but decreased
> bandwidth for downloading.  But the download frequency is very low - only when
> someone installs the UIMA Eclipse plugins into a new Eclipse instance.  I think
> pack200 was envisioned to help websites where the fetch was more frequent, and
> web-page load times were important. 
>
> Should we abandon doing pack200 packaging?
>
> Well, those 8 topics should get us started.  Perhaps it will be best to start a
> new thread for each one.
> I'm looking forward to the discussion. 
>
> Cheers. -Marshall
>

Re: redesigining our Eclipse plugin release build flow - list of issues to find out about

[Marshall Schor <ms...@schor.com>]

one more item:

We could change how we release Eclipse plugins, and break the version lock-step
which causes each version to release all the plugins, even when no change has
happened in the plugins (as is often the case!)

This would reduce the new JAR signings, by quite a bit, I think.

-Marshall


On 9/20/2017 1:18 PM, Marshall Schor wrote:
> one more "topic" to consider:
>
> If we abandon pack200 for compression, we could still use it for repacking, or
> JAR "normalization".
>
> pack200 has 2 phases; the first does things which change the JAR content/structure:
> 1. merges/sorts constant-ppol data in the class files and co-locates them
> 2. removes redundant class attributes
> 3. stores internal data structures
>
> The 2nd phase uses compression technologies to pack.
>
> The first phase needs to be done (using the -repack option) before signing; see
> http://docs.oracle.com/javase/1.5.0/docs/guide/deployment/deployment-guide/pack200.html
>
> It works like this:  pack (changing structure) -> unpack -> sign -> pack (no
> change to structure) -> deploy
>
> The first pack does the optimizations that change the structure; the 2nd one
> doesn't because the structures are already in the right way.
>
> I'm guessing not too much saving happens in normal JARs due to normalization,
> alone, but might be worth doing for big jars (like the eclipse uima runtime
> plugin - 4m).
>
> -Marshall
>
>
>
> On 9/20/2017 11:34 AM, Marshall Schor wrote:
>> I'm starting to think about redoing our Eclipse Plugin release flow.  I'm hoping
>> to start a thread where people can post answers to issues (there are many...).
>>
>> Motivation: start using the Symantec signing process for Jars (in addition to
>> the PGP Apache signing process), so that Eclipse, when installing new plugins
>> doesn't warn about unsigned artifacts.
>>
>> There are many things to figure out.  One is how to manage the cost of the
>> signing. Signing costs real money, which Apache covers (see
>> https://reference.apache.org/pmc/codesigning). 
>>
>> 1) To find out:  is the cost per Jar, or per "signing event" which can include
>> multiple Jars?  This can affect some design choices.
>>
>> 2) Another thing is when (in our release flow) to sign.  The signing process
>> alters the JARs.
>>
>> Our current flow is  1) generate candidate with PGP signature, 2) Vote  3)
>> repeat if failed rc, otherwise promote.  An Apache principle is to have the Vote
>> be on the signed artifacts, which are then promoted without modification.  This
>> implies the step 1) would need to include signing. 
>>
>> This means that every failed release candidate costs money.  I found out that
>> some projects "accept" this, but have low failed percentages, like 1 in 10.  In
>> our process, we seem to frequently have multiple release candidates per release. 
>>
>> Perhaps a way forward is to have a process for the Eclipse plugin release which
>> is different, and isn't invoked until a "preliminary" candidate review has been
>> done; once this has happened, and there's confidence that the signed version
>> would likely pass the vote, an official Eclipse plugin release candidate could
>> be posted with signed JARs.
>>
>> 3) Another thing to figure out, assuming that signing costs "per jar", is to
>> insure only new Jars are signed, not the older versions.  This could affect
>> efforts to limit the number of JARs (or not).
>>
>> 4) Another thing to figure out is what needs to be signed.  I think that only
>> Eclipse plugin jars need signing, and not the other jars associated with the
>> update site, like artifacts.jar, content.jar, and for each feature,
>> feat-name.jar (e.g. org.apache.uima.runtime_3.0.0.alpha02.jar).  But maybe
>> that's wrong.  The site https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F
>> says multiple things: 1) "by default, every Jar pushed to an update site is
>> signed", 2) "in standalone zip distributions, all JARed plugins will be
>> signed".  So, my guess is that you don't need to sign the other kinds of JARs. 
>>
>> Some websites imply that JARs other than the plugin ones are signed (
>> https://wiki.eclipse.org/Platform-releng-signedbuild ), but it's unclear if they
>> need signing in order to get the Eclipse install program not to complain.  That
>> website says the master zip having all the plugins and features is sent to the
>> signing process.
>>
>> 5) Some of our Jars (typically, the runtime Jars) have other Jars inside them. 
>> This site https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F says "JARs
>> nested at arbitrary depth within other JARs" are signed, too.  Is that needed to
>> make the Eclipse install program not complain about installing unsigned artifacts?
>>
>> 6) One site ( https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F ) says
>> there may be legal reasons you are not permitted to sign an embedded JAR.  I've
>> never seen anything about that, and don't know how to determine if such a
>> prohibition exists.  This page says, under what gets signed, "Some JARs may be
>> excluded if there are technical or legal reasons why they cannot be signed".
>>
>> 7) Signing and pack200 interact.  This page
>> https://wiki.eclipse.org/Tycho/Pack200#Pack200_and_Signing describes a flow:
>> 1. pack200 operated to "normalize" - this is specified by the "repack" option
>> 2. sign
>> 3. pack200 operated to generate the compressed version
>> 4. p2 metadata generation / updating
>>
>> Our current eclipse plugin build flow runs pack200 with both the "-repack" and
>> "-pack" options.  I think this would need to be split into two steps (assuming
>> we keep pack200 - see following), with the results from -repack being sent for
>> signing, and then the result subsequently packed.
>>
>> 8) I question if we need to do pack200 packing.  It has these effects:
>> 1. for every Jar, it generates 2 Jars: the original and the packed.  So there
>> are 2 jars that need "signing", thus it costs twice as much. 
>> 2. A typical Jar (caseditor.ide) is 69K bytes, packed 28K.  A big jar
>> uima-runtime is 4M, packed: 1.3M
>> So the result is increased space on the distribution medium, but decreased
>> bandwidth for downloading.  But the download frequency is very low - only when
>> someone installs the UIMA Eclipse plugins into a new Eclipse instance.  I think
>> pack200 was envisioned to help websites where the fetch was more frequent, and
>> web-page load times were important. 
>>
>> Should we abandon doing pack200 packaging?
>>
>> Well, those 8 topics should get us started.  Perhaps it will be best to start a
>> new thread for each one.
>> I'm looking forward to the discussion. 
>>
>> Cheers. -Marshall
>>
>

Re: redesigining our Eclipse plugin release build flow - list of issues to find out about

[Marshall Schor <ms...@schor.com>]

one more "topic" to consider:

If we abandon pack200 for compression, we could still use it for repacking, or
JAR "normalization".

pack200 has 2 phases; the first does things which change the JAR content/structure:
1. merges/sorts constant-ppol data in the class files and co-locates them
2. removes redundant class attributes
3. stores internal data structures

The 2nd phase uses compression technologies to pack.

The first phase needs to be done (using the -repack option) before signing; see
http://docs.oracle.com/javase/1.5.0/docs/guide/deployment/deployment-guide/pack200.html

It works like this:  pack (changing structure) -> unpack -> sign -> pack (no
change to structure) -> deploy

The first pack does the optimizations that change the structure; the 2nd one
doesn't because the structures are already in the right way.

I'm guessing not too much saving happens in normal JARs due to normalization,
alone, but might be worth doing for big jars (like the eclipse uima runtime
plugin - 4m).

-Marshall



On 9/20/2017 11:34 AM, Marshall Schor wrote:
> I'm starting to think about redoing our Eclipse Plugin release flow.  I'm hoping
> to start a thread where people can post answers to issues (there are many...).
>
> Motivation: start using the Symantec signing process for Jars (in addition to
> the PGP Apache signing process), so that Eclipse, when installing new plugins
> doesn't warn about unsigned artifacts.
>
> There are many things to figure out.  One is how to manage the cost of the
> signing. Signing costs real money, which Apache covers (see
> https://reference.apache.org/pmc/codesigning). 
>
> 1) To find out:  is the cost per Jar, or per "signing event" which can include
> multiple Jars?  This can affect some design choices.
>
> 2) Another thing is when (in our release flow) to sign.  The signing process
> alters the JARs.
>
> Our current flow is  1) generate candidate with PGP signature, 2) Vote  3)
> repeat if failed rc, otherwise promote.  An Apache principle is to have the Vote
> be on the signed artifacts, which are then promoted without modification.  This
> implies the step 1) would need to include signing. 
>
> This means that every failed release candidate costs money.  I found out that
> some projects "accept" this, but have low failed percentages, like 1 in 10.  In
> our process, we seem to frequently have multiple release candidates per release. 
>
> Perhaps a way forward is to have a process for the Eclipse plugin release which
> is different, and isn't invoked until a "preliminary" candidate review has been
> done; once this has happened, and there's confidence that the signed version
> would likely pass the vote, an official Eclipse plugin release candidate could
> be posted with signed JARs.
>
> 3) Another thing to figure out, assuming that signing costs "per jar", is to
> insure only new Jars are signed, not the older versions.  This could affect
> efforts to limit the number of JARs (or not).
>
> 4) Another thing to figure out is what needs to be signed.  I think that only
> Eclipse plugin jars need signing, and not the other jars associated with the
> update site, like artifacts.jar, content.jar, and for each feature,
> feat-name.jar (e.g. org.apache.uima.runtime_3.0.0.alpha02.jar).  But maybe
> that's wrong.  The site https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F
> says multiple things: 1) "by default, every Jar pushed to an update site is
> signed", 2) "in standalone zip distributions, all JARed plugins will be
> signed".  So, my guess is that you don't need to sign the other kinds of JARs. 
>
> Some websites imply that JARs other than the plugin ones are signed (
> https://wiki.eclipse.org/Platform-releng-signedbuild ), but it's unclear if they
> need signing in order to get the Eclipse install program not to complain.  That
> website says the master zip having all the plugins and features is sent to the
> signing process.
>
> 5) Some of our Jars (typically, the runtime Jars) have other Jars inside them. 
> This site https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F says "JARs
> nested at arbitrary depth within other JARs" are signed, too.  Is that needed to
> make the Eclipse install program not complain about installing unsigned artifacts?
>
> 6) One site ( https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F ) says
> there may be legal reasons you are not permitted to sign an embedded JAR.  I've
> never seen anything about that, and don't know how to determine if such a
> prohibition exists.  This page says, under what gets signed, "Some JARs may be
> excluded if there are technical or legal reasons why they cannot be signed".
>
> 7) Signing and pack200 interact.  This page
> https://wiki.eclipse.org/Tycho/Pack200#Pack200_and_Signing describes a flow:
> 1. pack200 operated to "normalize" - this is specified by the "repack" option
> 2. sign
> 3. pack200 operated to generate the compressed version
> 4. p2 metadata generation / updating
>
> Our current eclipse plugin build flow runs pack200 with both the "-repack" and
> "-pack" options.  I think this would need to be split into two steps (assuming
> we keep pack200 - see following), with the results from -repack being sent for
> signing, and then the result subsequently packed.
>
> 8) I question if we need to do pack200 packing.  It has these effects:
> 1. for every Jar, it generates 2 Jars: the original and the packed.  So there
> are 2 jars that need "signing", thus it costs twice as much. 
> 2. A typical Jar (caseditor.ide) is 69K bytes, packed 28K.  A big jar
> uima-runtime is 4M, packed: 1.3M
> So the result is increased space on the distribution medium, but decreased
> bandwidth for downloading.  But the download frequency is very low - only when
> someone installs the UIMA Eclipse plugins into a new Eclipse instance.  I think
> pack200 was envisioned to help websites where the fetch was more frequent, and
> web-page load times were important. 
>
> Should we abandon doing pack200 packaging?
>
> Well, those 8 topics should get us started.  Perhaps it will be best to start a
> new thread for each one.
> I'm looking forward to the discussion. 
>
> Cheers. -Marshall
>

Re: redesigining our Eclipse plugin release build flow - list of issues to find out about

[Peter Klügl <pe...@averbis.com>]

Kudos to you for approaching this.


I'll try to help where I can, but I am very committed to other things
right now :-(

(Maybe in two weeks, I'll have more time)


Best,


Peter


Am 20.09.2017 um 17:34 schrieb Marshall Schor:
> I'm starting to think about redoing our Eclipse Plugin release flow.  I'm hoping
> to start a thread where people can post answers to issues (there are many...).
>
> Motivation: start using the Symantec signing process for Jars (in addition to
> the PGP Apache signing process), so that Eclipse, when installing new plugins
> doesn't warn about unsigned artifacts.
>
> There are many things to figure out.  One is how to manage the cost of the
> signing. Signing costs real money, which Apache covers (see
> https://reference.apache.org/pmc/codesigning). 
>
> 1) To find out:  is the cost per Jar, or per "signing event" which can include
> multiple Jars?  This can affect some design choices.
>
> 2) Another thing is when (in our release flow) to sign.  The signing process
> alters the JARs.
>
> Our current flow is  1) generate candidate with PGP signature, 2) Vote  3)
> repeat if failed rc, otherwise promote.  An Apache principle is to have the Vote
> be on the signed artifacts, which are then promoted without modification.  This
> implies the step 1) would need to include signing. 
>
> This means that every failed release candidate costs money.  I found out that
> some projects "accept" this, but have low failed percentages, like 1 in 10.  In
> our process, we seem to frequently have multiple release candidates per release. 
>
> Perhaps a way forward is to have a process for the Eclipse plugin release which
> is different, and isn't invoked until a "preliminary" candidate review has been
> done; once this has happened, and there's confidence that the signed version
> would likely pass the vote, an official Eclipse plugin release candidate could
> be posted with signed JARs.
>
> 3) Another thing to figure out, assuming that signing costs "per jar", is to
> insure only new Jars are signed, not the older versions.  This could affect
> efforts to limit the number of JARs (or not).
>
> 4) Another thing to figure out is what needs to be signed.  I think that only
> Eclipse plugin jars need signing, and not the other jars associated with the
> update site, like artifacts.jar, content.jar, and for each feature,
> feat-name.jar (e.g. org.apache.uima.runtime_3.0.0.alpha02.jar).  But maybe
> that's wrong.  The site https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F
> says multiple things: 1) "by default, every Jar pushed to an update site is
> signed", 2) "in standalone zip distributions, all JARed plugins will be
> signed".  So, my guess is that you don't need to sign the other kinds of JARs. 
>
> Some websites imply that JARs other than the plugin ones are signed (
> https://wiki.eclipse.org/Platform-releng-signedbuild ), but it's unclear if they
> need signing in order to get the Eclipse install program not to complain.  That
> website says the master zip having all the plugins and features is sent to the
> signing process.
>
> 5) Some of our Jars (typically, the runtime Jars) have other Jars inside them. 
> This site https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F says "JARs
> nested at arbitrary depth within other JARs" are signed, too.  Is that needed to
> make the Eclipse install program not complain about installing unsigned artifacts?
>
> 6) One site ( https://wiki.eclipse.org/JAR_Signing#What_gets_signed.3F ) says
> there may be legal reasons you are not permitted to sign an embedded JAR.  I've
> never seen anything about that, and don't know how to determine if such a
> prohibition exists.  This page says, under what gets signed, "Some JARs may be
> excluded if there are technical or legal reasons why they cannot be signed".
>
> 7) Signing and pack200 interact.  This page
> https://wiki.eclipse.org/Tycho/Pack200#Pack200_and_Signing describes a flow:
> 1. pack200 operated to "normalize" - this is specified by the "repack" option
> 2. sign
> 3. pack200 operated to generate the compressed version
> 4. p2 metadata generation / updating
>
> Our current eclipse plugin build flow runs pack200 with both the "-repack" and
> "-pack" options.  I think this would need to be split into two steps (assuming
> we keep pack200 - see following), with the results from -repack being sent for
> signing, and then the result subsequently packed.
>
> 8) I question if we need to do pack200 packing.  It has these effects:
> 1. for every Jar, it generates 2 Jars: the original and the packed.  So there
> are 2 jars that need "signing", thus it costs twice as much. 
> 2. A typical Jar (caseditor.ide) is 69K bytes, packed 28K.  A big jar
> uima-runtime is 4M, packed: 1.3M
> So the result is increased space on the distribution medium, but decreased
> bandwidth for downloading.  But the download frequency is very low - only when
> someone installs the UIMA Eclipse plugins into a new Eclipse instance.  I think
> pack200 was envisioned to help websites where the fetch was more frequent, and
> web-page load times were important. 
>
> Should we abandon doing pack200 packaging?
>
> Well, those 8 topics should get us started.  Perhaps it will be best to start a
> new thread for each one.
> I'm looking forward to the discussion. 
>
> Cheers. -Marshall