3rd party bundled binaries - what can we do and cannot do?

[Lars Bruun-Hansen <lb...@gmail.com>]

Hi all,

(and possibly mentors in particular)

As a module reviewer I'm uncertain/confused about how to deal with 3rd
party libraries.

Here's how I understand it:  An Apache release need to comply with certain
standards, one of them being that any bundled 3rd party binary must be duly
vetted. We can't have a build process where we collect 3rd party binaries
from just about any location on the internet. So far so good. Understood.
But what does it mean practically?

Reading the various e-mails it seems that for our release process to be
approved, we must limit the location from where we fetch 3rd party libs to
the following:

* Apache's own Maven repo
* Maven Central, but only if the project of the 3rd party lib is an Apache
project.
* The project's VCS  (we are trying to avoid this solution)
* A location fully under the project's control, e.g. like
hg.netbeans.org/binaries.

Note the things NOT on the list.

Can anyone summarize/conclude here?

Thx

Lars

Re: 3rd party bundled binaries - what can we do and cannot do?

[Antonio Vieiro <an...@vieiro.net>]

Thanks, Bertrand,

I’ll update my email compiler with that :-D

Un abrazo,
Antonio

> El 2 nov 2017, a las 15:19, Bertrand Delacretaz <bd...@apache.org> escribió:
> 
> Hi Antonio,
> 
> On Thu, Nov 2, 2017 at 2:58 PM, Antonio Vieiro <an...@vieiro.net> wrote:
>> ....I think it could be a good idea to talk to those “release reviewers” and ask them what the possible problems could be...
> 
> You cannot do that in the abstract.
> 
> Those things are grey areas, people might just be unhappy about them
> or they might pull a valid reason for rejecting a release based on
> those, that I didn't think of. Not likely but not impossible either.
> In principle such things should be acceptable for a project in
> incubation but that's just my opinion.
> 
> During incubation, every Incubator PMC member is potentially a release
> reviewer. I think that's a few hundred people.
> 
> So best is to prepare a release, ideally with justifications for
> getting binaries from any other places than Maven Central during
> build, and put it to a vote.
> 
> -Bertrand

Re: 3rd party bundled binaries - what can we do and cannot do?

[Bertrand Delacretaz <bd...@apache.org>]

Hi Antonio,

On Thu, Nov 2, 2017 at 2:58 PM, Antonio Vieiro <an...@vieiro.net> wrote:
> ....I think it could be a good idea to talk to those “release reviewers” and ask them what the possible problems could be...

You cannot do that in the abstract.

Those things are grey areas, people might just be unhappy about them
or they might pull a valid reason for rejecting a release based on
those, that I didn't think of. Not likely but not impossible either.
In principle such things should be acceptable for a project in
incubation but that's just my opinion.

During incubation, every Incubator PMC member is potentially a release
reviewer. I think that's a few hundred people.

So best is to prepare a release, ideally with justifications for
getting binaries from any other places than Maven Central during
build, and put it to a vote.

-Bertrand

Re: 3rd party bundled binaries - what can we do and cannot do?

[Antonio Vieiro <an...@vieiro.net>]

Hi,


> El 2 nov 2017, a las 10:15, Bertrand Delacretaz <bd...@apache.org> escribió:
> 
> I don't think that's a hard requirement, but grabbing binaries from
> non-standard places is not a good thing in general and might raise
> questions from release reviewers.
> 
> However, IIUC the NetBeans build strongly checks the digests of all
> binaries that it uses, which makes it less of a problem.

Bertrand, these last two paragraphs broke my email compiler with the following errors/warnings: :-)

- “non-standard places” and “release reviewers” are undefined.
- the compiler also raised warnings while parsing “I don’t think”, “might raise questions” and “less of a problem”, stating that those were ambiguous.

Now, seriously, I think it could be a good idea to talk to those “release reviewers” and ask them what the possible problems could be. 

That way we could start working on a solution that avoids those “less problems” in the future, as well as the one stated in your last paragraph, here:

> 
> Downloading binaries from locations which have an uncertain future
> (like hg.netbeans.org/binaries IIUC) means people might be unable to
> build NetBeans in the future, and that's not good, just from a
> practical and sustainability standpoint - so might be frowned upon as
> well, but less so for a first incubating release.
> 
> -Bertrand


Un abrazo,
Antonio

Re: 3rd party bundled binaries - what can we do and cannot do?

[Jan Lahoda <la...@gmail.com>]

On Thu, Nov 2, 2017 at 10:15 AM, Bertrand Delacretaz <bdelacretaz@apache.org
> wrote:

> Hi,
>
> On Thu, Nov 2, 2017 at 9:39 AM, Lars Bruun-Hansen
> <lb...@gmail.com> wrote:
> > ....it seems that for our release process to be
> > approved, we must limit the location from where we fetch 3rd party libs
> to
> > the following...
>
> I don't think that's a hard requirement, but grabbing binaries from
> non-standard places is not a good thing in general and might raise
> questions from release reviewers.
>
> However, IIUC the NetBeans build strongly checks the digests of all
> binaries that it uses, which makes it less of a problem.
>
> Downloading binaries from locations which have an uncertain future
> (like hg.netbeans.org/binaries IIUC) means people might be unable to
> build NetBeans in the future, and that's not good, just from a
> practical and sustainability standpoint - so might be frowned upon as
> well, but less so for a first incubating release.
>

While I agree the dependency on /binaries is not ideal, it is AFAIK
possible change the location when building to some other location, or fill
the local cache manually. Still troublesome, but better IMO to try to
release and find out about other problematic things we don't know about yet.

I'd say we have dependencies of several kinds:
-non-problematic (OK license, are on maven), e.g. Apache Felix, JNA
-not really problematic, but will take time, e.g. launchers(*)
-somewhat problematic (not on maven, etc.), like Jemmy (the version that NB
is marked as CDDLGPLv2CPE as far as I can tell, is not on maven, new
versions are GPLv2CPE AFAIK)
-more problematic, like libs.javafx (compile-time only)
-very problematic (libs.javacapi, libs.javaimpl)

Specifically for the launchers, the situation is like this: on Windows,
NetBeans (platform, platform applications and IDE) need an exe launcher.
Currently, AFAIK, the (standard Java) build is not requiring any C tool
chain, and produced build are mostly platform independent. I.e. even a
Linux build will produce a binary useable on Windows (with the exe
launchers). I believe it would be very good to keep these properties.
Currently, the build is downloading the original CDDL+GPLv2CPE launchers
from /binaries. My understanding is that the following approach would
provide a solution (but is likely to take some time):
-make a standalone release of (just) the launchers (sources)
-build convenience binaries from them and upload them somewhere (ideally
maven)
-the platform/IDE build would download and use these convenience binaries

(We would need to do the same for a few more native binaries, like the
profiler libraries.)

Jan


> -Bertrand
>

Re: 3rd party bundled binaries - what can we do and cannot do?

[Lars Bruun-Hansen <lb...@gmail.com>]

Ok, thanks.  Then I've misinterpreted which is good.

I was trying to avoid the situation where the release process wouldn't be
approved by release reviewers and we would have to re-do work already done.
(I'm lazy by nature).  It seems this isn't written in stone and it is up to
the reviewers.

Quote from Jaroslav Tulach about his experience:

"On the other hand during review of HTML/Java API I had to remove download
from Google Maven repository - it was seen as untrusted. I assume the same
will be said about the eclipse repository."

And I assume that that the HTML/Java API 3rd party lib retrieval process
uses digests too.

In any case it would be nice if we could have some kind assurance that what
we do will eventually be approved. Understand fully that this is a grey
area. If this is so, then my suggestion is that we - for now - stay on a
very narrow path, pretty much as per my list. Then we are guaranteed it
will not be an issue in the release review process.

Overly cautious?  Possibly, but I value getting a release out VERY SOON
above just about everything else. Hence my suggestion. Perhaps others see
it differently.

my 2c

Lars



On Thu, Nov 2, 2017 at 10:15 AM, Bertrand Delacretaz <bdelacretaz@apache.org
> wrote:

> Hi,
>
> On Thu, Nov 2, 2017 at 9:39 AM, Lars Bruun-Hansen
> <lb...@gmail.com> wrote:
> > ....it seems that for our release process to be
> > approved, we must limit the location from where we fetch 3rd party libs
> to
> > the following...
>
> I don't think that's a hard requirement, but grabbing binaries from
> non-standard places is not a good thing in general and might raise
> questions from release reviewers.
>
> However, IIUC the NetBeans build strongly checks the digests of all
> binaries that it uses, which makes it less of a problem.
>
> Downloading binaries from locations which have an uncertain future
> (like hg.netbeans.org/binaries IIUC) means people might be unable to
> build NetBeans in the future, and that's not good, just from a
> practical and sustainability standpoint - so might be frowned upon as
> well, but less so for a first incubating release.
>
> -Bertrand
>

Re: 3rd party bundled binaries - what can we do and cannot do?

[Bertrand Delacretaz <bd...@apache.org>]

Hi,

On Thu, Nov 2, 2017 at 9:39 AM, Lars Bruun-Hansen
<lb...@gmail.com> wrote:
> ....it seems that for our release process to be
> approved, we must limit the location from where we fetch 3rd party libs to
> the following...

I don't think that's a hard requirement, but grabbing binaries from
non-standard places is not a good thing in general and might raise
questions from release reviewers.

However, IIUC the NetBeans build strongly checks the digests of all
binaries that it uses, which makes it less of a problem.

Downloading binaries from locations which have an uncertain future
(like hg.netbeans.org/binaries IIUC) means people might be unable to
build NetBeans in the future, and that's not good, just from a
practical and sustainability standpoint - so might be frowned upon as
well, but less so for a first incubating release.

-Bertrand