You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@netbeans.apache.org by Jaroslav Tulach <ja...@oracle.com> on 2016/12/08 09:14:09 UTC
Checksum was: Download binaries from Maven
Emilian wrote:
> > I wouldn't call this "proprietary". It's basically a lib/ folder stored an
> > a public HTTP server. Perhaps 'non-standard' would be the right word but
> > it's a very simple, open and functional system.
The slight trouble is that we need our place to host it. That is going to be
resolved if we use existing Maven repositories.
On čtvrtek 8. prosince 2016 9:28:14 CET Niclas Hedhman wrote:
> Maven repository normally contains hashes for each file. They are intended
> for download integrity check, but you could also keep the hash in your
> distro.
Right, there are SHA1 and MD5 checksums next to the JAR artifact - I can
modify our download code to download one of them and use it for the
consistency of downloaded bits check. Acceptable?
Btw. as the log from
http://deadlock.netbeans.org/job/prototypes-MavenDownload269264/2/console
shows the download seems to be running fine:
[downloadbinaries] Creating /space/hudson_slave/workspace/prototypes-
MavenDownload269264/javahelp/external/jhall-2.0_05.jar
[downloadbinaries] Downloading: http://central.maven.org/maven2/junit/junit/
4.12/junit-4.12.jar
[downloadbinaries] Downloading: http://central.maven.org/maven2/org/hamcrest/
hamcrest-core/1.3/hamcrest-core-1.3.jar
JUnit & co. have just been downloaded from Maven central.
> Additionally, many projects uploads detached PGP signatures of each
> file, for you to truly verify its origin if you so desire.
Yes, security is important. I guess I should at least use "https" when
connecting to Maven central, so we make man-in-middle attack harder...
> On Thu, Dec 8, 2016 at 2:13 AM, Emilian Bold <em...@apache.org> wrote:
> > The previous system was a bit more solid if only because you also saved
> > hash in the dependency. So after download you could do an automated
> > integrity check.
> >
> > The Maven groupId:artifactId:version does not guarantee you have the same
> > file. Perhaps groupId:artifactId:version:hash should be used instead?
That is possible as well. However people using Maven are fine with pom.xml
(without hashes) which works reliably. I hope that if we mimic that, we'll be
fine as well.
> > Also, is there some (automatic) check during this transition that verifies
> > the binary JAR from http://hg.netbeans.org/binaries and the replacing one
> > from the Maven repository are actually identical?
No, there is none. I assume the changes to binaries-list files to be done
manually and the check done by human.
-jt
Re: Checksum was: Download binaries from Maven
Posted by Emilian Bold <em...@gmail.com>.
SHA-256 seems to be the most widely used hash.
It doesn't matter if it will be found weak in the future, we can always
switch to something better.
HTTPS would also be recommended, but in addition to having trusted hashes,
not as a replacement.
--emi
On Thu, Dec 8, 2016 at 2:50 PM, Jaroslav Tulach <ja...@oracle.com>
wrote:
> Hello Bertrand,
> thanks for your insight.
>
> On čtvrtek 8. prosince 2016 10:35:55 CET Bertrand Delacretaz wrote:
> > > ...there are SHA1 and MD5 checksums next to the JAR artifact - I can
> > > modify our download code to download one of them and use it for the
> > > consistency of downloaded bits check....
> >
> > To be useful the digests need to come from a trusted source.
>
> I see. I am starting to believe Emilian is right...
>
> > For Apache releases, as described at
> > http://www.apache.org/dev/release-publishing.html the digests, keys
> > etc. must come from ASF hosts. This allows the release archives to be
> > downloaded via channels that don't need to be trusted.
>
> ...if we keep the digests in the source code (as NetBeans do currently), we
> make them "absolutely trusted". Then we can continue to download from Maven
> central as we do now and everything remains kosher.
>
> > Although we do trust Maven Central in general, if you're designing a
> > new mechanism I suggest doing the same - distribute the digests from
> > https://dist.apache.org/repos/dist/release/ (*) and the binaries from
> > wherever.
>
> I don't get meaning of this paragraph, but assuming we have the digests
> inside
> of the source code, we don't have to worry about this. I guess.
>
> > Note also that MD5 shouldn't be trusted anymore, see
> > natmchugh.blogspot.com/2014/10/how-i-created-two-images-
> with-same-md5.html
> > for example.
>
> What digest to use then? SHA1? Or connect to the Maven repositories via
> HTTPS?
>
> I believe the idea of digest was never to fully shield anyone from
> maliciously
> prepared files - the idea that one can uniquely identify any file in the
> Universe by 32 bytes sounds ridiculously silly, doesn't it? - rather to
> shield
> one against randomly broken bits.
>
> Regardless of what digest algorithm we use, one day it will be cracked
> anyway.
> Thus we can choose any (incuding poor MD5), but rather complicate the
> man-in-
> middle attack by using HTTPS (which is using its own certificates) for
> download
> - that will multiply the improbability of a hack more than choosing SHA1 or
> SHA7, SHA337, etc., in my opinion.
>
> -jt
>
>
Re: Checksum was: Download binaries from Maven
Posted by Bertrand Delacretaz <bd...@apache.org>.
Hi Jaroslav,
On Thu, Dec 8, 2016 at 1:50 PM, Jaroslav Tulach
<ja...@oracle.com> wrote:
> ...if we keep the digests in the source code (as NetBeans do currently), we
> make them "absolutely trusted". Then we can continue to download from Maven
> central as we do now and everything remains kosher....
Ok - I'm clueless about how that works for NetBeans so not sure what
"in the source code" means here.
I assume the use case is downloading modules from a NetBeans instance
running on my own computer, where does that instance get the digests
then? Hardcoded within itself? Via HTTP from the source code
repository? The latter wouldn't be acceptable in an Apache context,
the distribution channel needs to be agreed upon with ASF infra.
> ...What digest to use then? SHA1? Or connect to the Maven repositories via HTTPS?
I'm not a crypto specialist but I think SHA-256 is a reasonable choice nowadays.
-Bertrand
Re: Checksum was: Download binaries from Maven
Posted by Jaroslav Tulach <ja...@oracle.com>.
Hello Bertrand,
thanks for your insight.
On čtvrtek 8. prosince 2016 10:35:55 CET Bertrand Delacretaz wrote:
> > ...there are SHA1 and MD5 checksums next to the JAR artifact - I can
> > modify our download code to download one of them and use it for the
> > consistency of downloaded bits check....
>
> To be useful the digests need to come from a trusted source.
I see. I am starting to believe Emilian is right...
> For Apache releases, as described at
> http://www.apache.org/dev/release-publishing.html the digests, keys
> etc. must come from ASF hosts. This allows the release archives to be
> downloaded via channels that don't need to be trusted.
...if we keep the digests in the source code (as NetBeans do currently), we
make them "absolutely trusted". Then we can continue to download from Maven
central as we do now and everything remains kosher.
> Although we do trust Maven Central in general, if you're designing a
> new mechanism I suggest doing the same - distribute the digests from
> https://dist.apache.org/repos/dist/release/ (*) and the binaries from
> wherever.
I don't get meaning of this paragraph, but assuming we have the digests inside
of the source code, we don't have to worry about this. I guess.
> Note also that MD5 shouldn't be trusted anymore, see
> natmchugh.blogspot.com/2014/10/how-i-created-two-images-with-same-md5.html
> for example.
What digest to use then? SHA1? Or connect to the Maven repositories via HTTPS?
I believe the idea of digest was never to fully shield anyone from maliciously
prepared files - the idea that one can uniquely identify any file in the
Universe by 32 bytes sounds ridiculously silly, doesn't it? - rather to shield
one against randomly broken bits.
Regardless of what digest algorithm we use, one day it will be cracked anyway.
Thus we can choose any (incuding poor MD5), but rather complicate the man-in-
middle attack by using HTTPS (which is using its own certificates) for download
- that will multiply the improbability of a hack more than choosing SHA1 or
SHA7, SHA337, etc., in my opinion.
-jt
Re: Checksum was: Download binaries from Maven
Posted by Emilian Bold <em...@gmail.com>.
Jaroslav, I noticed you reverted some of the external file changes to
download from the NetBeans server instead of the maven repo due to
proxy issue.
I was wondering: wouldn't it be better to change the binaries-list
format to something like:
HASH FILENAME [mvn://G:A:V]
with the Maven part being optional?
This has multiple advantages:
* we make sure the hash is really the same once we move to Maven
* we could even add an optional flag to download from Maven or from
the NetBeans binaries server. Or maybe use one as a fallback for the
other, etc.
* we allow another name for the Maven artifact. I'm not certain, but
perhaps some JAR files do not have the proper Maven name. This way we
don't have to change any reference to the JAR.
--emi
On Wed, Dec 21, 2016 at 7:19 PM, Jaroslav Tulach
<ja...@oracle.com> wrote:
> Support for downloading binaries from Maven central during NetBeans build has
> just been propagated to main-silver repository: http://hg.netbeans.org/main-silver/rev/00281b17877a
>
> Merry Christmas!
> -jt
>
> On pátek 9. prosince 2016 10:16:45 CET Jaroslav Tulach wrote:
>> On pátek 9. prosince 2016 9:44:24 CET Mark Struberg wrote:
>> > We could split the 2 up.
>>
>> OK, thanks Mark, Bernard. I need your input to sort my thoughts as the
>> discussion seems to go in unexpected directions...
>>
>> > For ASF binaries we have *all* files hosted at ASF + the actual binaries
>> > are also distributed downstream to mirrors.
>>
>> Are we talking about the same thing? I care about binaries needed to build
>> NetBeans. Here is my change, so we get better understanding what I am trying
>> to do:
>>
>> http://hg.netbeans.org/ergonomics/rev/4270da1cffd4
>>
>> For example I need junit.jar, where shall I download it from? It isn't ASF
>> binary, right? So where it should come from?
>>
>> I am proposing to download it from Maven repository, but I am fine with any
>> other alternative.
>> -jt
>
>
Re: Checksum was: Download binaries from Maven
Posted by Jaroslav Tulach <ja...@oracle.com>.
Support for downloading binaries from Maven central during NetBeans build has
just been propagated to main-silver repository: http://hg.netbeans.org/main-silver/rev/00281b17877a
Merry Christmas!
-jt
On pátek 9. prosince 2016 10:16:45 CET Jaroslav Tulach wrote:
> On pátek 9. prosince 2016 9:44:24 CET Mark Struberg wrote:
> > We could split the 2 up.
>
> OK, thanks Mark, Bernard. I need your input to sort my thoughts as the
> discussion seems to go in unexpected directions...
>
> > For ASF binaries we have *all* files hosted at ASF + the actual binaries
> > are also distributed downstream to mirrors.
>
> Are we talking about the same thing? I care about binaries needed to build
> NetBeans. Here is my change, so we get better understanding what I am trying
> to do:
>
> http://hg.netbeans.org/ergonomics/rev/4270da1cffd4
>
> For example I need junit.jar, where shall I download it from? It isn't ASF
> binary, right? So where it should come from?
>
> I am proposing to download it from Maven repository, but I am fine with any
> other alternative.
> -jt
Re: Checksum in the repository
Posted by Emilian Bold <em...@gmail.com>.
Looks good to me.
În vin., 9 dec. 2016 la 14:33 Jaroslav Tulach <ja...@oracle.com>
a scris:
> Hi guys,
> here is implementation of what we agreed to:
> http://hg.netbeans.org/ergonomics/rev/4d2653904feb
>
> -jt
>
> On pátek 9. prosince 2016 12:09:07 CET Emilian Bold wrote:
> > > And it's also unfortunately common to omit digests when specifying
> which
> > binaries are needed, as AFAIK the Maven pom and dependencies mechanisms
> > don't allow that.
> >
> > Yes, but since this code is still being written, it's trivial for us to
> > *continue* saving hashes for dependencies.
> >
> > I mean, the previous dependency download code used hashes for files
> > downloaded from a NetBeans-controlled server.
> >
> > Why would we skip this now that binaries are hosted on 3rd party
> servers!?
> >
> > The fact that Maven still does not support this is a glaring (security)
> bug
> > imho.
> >
> >
> >
> > --emi
> >
> > On Fri, Dec 9, 2016 at 11:40 AM, Bertrand Delacretaz <
> bdelacretaz@apache.org
> > > wrote:
> > >
> > > On Fri, Dec 9, 2016 at 10:16 AM, Jaroslav Tulach
> > >
> > > <ja...@oracle.com> wrote:
> > > > ...Are we talking about the same thing? I care about binaries needed
> to
> > >
> > > build
> > >
> > > > NetBeans....
> > >
> > > Ahhhh, so it looks like I was totally confused, sorry about that. I
> > > thought this was about NetBeans downloading plugins at runtime.
> > >
> > > It's common for Apache projects to get binaries from a Maven
> > > repository at build time.
> > >
> > > And it's also unfortunately common to omit digests when specifying
> > > which binaries are needed, as AFAIK the Maven pom and dependencies
> > > mechanisms don't allow that.
> > >
> > > I think the idea is that if someone cares they would get binaries from
> > > their own controlled Maven repository but I suspect few people do
> > > that.
> > >
> > > So it's fine for NetBeans to do the same and get dependencies at build
> > > time from a Maven repository using similar mechanisms than Maven pom
> > > dependencies. Including a possibility to configure the Maven
> > > repository URL.
> > >
> > > IIUC your changes NetBeans is using a custom ant task to retrieve
> > > dependencies, I suppose there's existing Ant tasks to do that with
> > > Maven. I haven't used Ant in ages so I cannot be more specific.
> > >
> > > -Bertrand
>
>
>
Re: Checksum in the repository
Posted by Jaroslav Tulach <ja...@oracle.com>.
Hi guys,
here is implementation of what we agreed to:
http://hg.netbeans.org/ergonomics/rev/4d2653904feb
-jt
On pátek 9. prosince 2016 12:09:07 CET Emilian Bold wrote:
> > And it's also unfortunately common to omit digests when specifying which
> binaries are needed, as AFAIK the Maven pom and dependencies mechanisms
> don't allow that.
>
> Yes, but since this code is still being written, it's trivial for us to
> *continue* saving hashes for dependencies.
>
> I mean, the previous dependency download code used hashes for files
> downloaded from a NetBeans-controlled server.
>
> Why would we skip this now that binaries are hosted on 3rd party servers!?
>
> The fact that Maven still does not support this is a glaring (security) bug
> imho.
>
>
>
> --emi
>
> On Fri, Dec 9, 2016 at 11:40 AM, Bertrand Delacretaz <bdelacretaz@apache.org
> > wrote:
> >
> > On Fri, Dec 9, 2016 at 10:16 AM, Jaroslav Tulach
> >
> > <ja...@oracle.com> wrote:
> > > ...Are we talking about the same thing? I care about binaries needed to
> >
> > build
> >
> > > NetBeans....
> >
> > Ahhhh, so it looks like I was totally confused, sorry about that. I
> > thought this was about NetBeans downloading plugins at runtime.
> >
> > It's common for Apache projects to get binaries from a Maven
> > repository at build time.
> >
> > And it's also unfortunately common to omit digests when specifying
> > which binaries are needed, as AFAIK the Maven pom and dependencies
> > mechanisms don't allow that.
> >
> > I think the idea is that if someone cares they would get binaries from
> > their own controlled Maven repository but I suspect few people do
> > that.
> >
> > So it's fine for NetBeans to do the same and get dependencies at build
> > time from a Maven repository using similar mechanisms than Maven pom
> > dependencies. Including a possibility to configure the Maven
> > repository URL.
> >
> > IIUC your changes NetBeans is using a custom ant task to retrieve
> > dependencies, I suppose there's existing Ant tasks to do that with
> > Maven. I haven't used Ant in ages so I cannot be more specific.
> >
> > -Bertrand
Re: Checksum was: Download binaries from Maven
Posted by Emilian Bold <em...@gmail.com>.
> And it's also unfortunately common to omit digests when specifying which
binaries are needed, as AFAIK the Maven pom and dependencies mechanisms
don't allow that.
Yes, but since this code is still being written, it's trivial for us to
*continue* saving hashes for dependencies.
I mean, the previous dependency download code used hashes for files
downloaded from a NetBeans-controlled server.
Why would we skip this now that binaries are hosted on 3rd party servers!?
The fact that Maven still does not support this is a glaring (security) bug
imho.
--emi
On Fri, Dec 9, 2016 at 11:40 AM, Bertrand Delacretaz <bdelacretaz@apache.org
> wrote:
> On Fri, Dec 9, 2016 at 10:16 AM, Jaroslav Tulach
> <ja...@oracle.com> wrote:
> > ...Are we talking about the same thing? I care about binaries needed to
> build
> > NetBeans....
>
> Ahhhh, so it looks like I was totally confused, sorry about that. I
> thought this was about NetBeans downloading plugins at runtime.
>
> It's common for Apache projects to get binaries from a Maven
> repository at build time.
>
> And it's also unfortunately common to omit digests when specifying
> which binaries are needed, as AFAIK the Maven pom and dependencies
> mechanisms don't allow that.
>
> I think the idea is that if someone cares they would get binaries from
> their own controlled Maven repository but I suspect few people do
> that.
>
> So it's fine for NetBeans to do the same and get dependencies at build
> time from a Maven repository using similar mechanisms than Maven pom
> dependencies. Including a possibility to configure the Maven
> repository URL.
>
> IIUC your changes NetBeans is using a custom ant task to retrieve
> dependencies, I suppose there's existing Ant tasks to do that with
> Maven. I haven't used Ant in ages so I cannot be more specific.
>
> -Bertrand
>
Re: Checksum was: Download binaries from Maven
Posted by Bertrand Delacretaz <bd...@apache.org>.
On Fri, Dec 9, 2016 at 10:16 AM, Jaroslav Tulach
<ja...@oracle.com> wrote:
> ...Are we talking about the same thing? I care about binaries needed to build
> NetBeans....
Ahhhh, so it looks like I was totally confused, sorry about that. I
thought this was about NetBeans downloading plugins at runtime.
It's common for Apache projects to get binaries from a Maven
repository at build time.
And it's also unfortunately common to omit digests when specifying
which binaries are needed, as AFAIK the Maven pom and dependencies
mechanisms don't allow that.
I think the idea is that if someone cares they would get binaries from
their own controlled Maven repository but I suspect few people do
that.
So it's fine for NetBeans to do the same and get dependencies at build
time from a Maven repository using similar mechanisms than Maven pom
dependencies. Including a possibility to configure the Maven
repository URL.
IIUC your changes NetBeans is using a custom ant task to retrieve
dependencies, I suppose there's existing Ant tasks to do that with
Maven. I haven't used Ant in ages so I cannot be more specific.
-Bertrand
Re: Checksum was: Download binaries from Maven
Posted by Jaroslav Tulach <ja...@oracle.com>.
On pátek 9. prosince 2016 9:44:24 CET Mark Struberg wrote:
> We could split the 2 up.
OK, thanks Mark, Bernard. I need your input to sort my thoughts as the
discussion seems to go in unexpected directions...
> For ASF binaries we have *all* files hosted at ASF + the actual binaries are
> also distributed downstream to mirrors.
Are we talking about the same thing? I care about binaries needed to build
NetBeans. Here is my change, so we get better understanding what I am trying
to do:
http://hg.netbeans.org/ergonomics/rev/4270da1cffd4
For example I need junit.jar, where shall I download it from? It isn't ASF
binary, right? So where it should come from?
I am proposing to download it from Maven repository, but I am fine with any
other alternative.
-jt
Re: Checksum was: Download binaries from Maven
Posted by Bertrand Delacretaz <bd...@apache.org>.
Hi,
On Fri, Dec 9, 2016 at 9:44 AM, Mark Struberg <st...@yahoo.de.invalid> wrote:
> ...even if a certain NetBeans plugin is GPLv2 then the asc and hashes are still perfectly
> hostable at the ASF I'd say....
I think this requires some caution, to make sure there's an open and
fair way to get one's digests in that master list.
Adding a plugin's digest is a form of endorsing, which could be abused
or could be made unfair depending on how and which digests are
accepted for addition.
But that's a different story, on the technical side I agree with what
you suggest.
-Bertrand
Re: Checksum was: Download binaries from Maven
Posted by Mark Struberg <st...@yahoo.de.INVALID>.
We could split the 2 up.
For ASF binaries we have *all* files hosted at ASF + the actual binaries are also distributed downstream to mirrors. The asc, md5 and sha1intentionally don't get mirrored as Bertrand explained.
For some NetBeans artifact this is a bit different, especially if they are not CategoryA licensed [1]. In that case we could distribute the binaries via maven or other location, but we could still have an index with all the asc, md5 and sha1 hosted at ASF. Because even if a certain NetBeans plugin is GPLv2 then the asc and hashes are still perfectly hostable at the ASF I'd say.
LieGrue,
strub
[1] https://www.apache.org/legal/resolved
> Am 08.12.2016 um 10:35 schrieb Bertrand Delacretaz <bd...@apache.org>:
>
> Hi,
>
> On Thu, Dec 8, 2016 at 10:14 AM, Jaroslav Tulach
> <ja...@oracle.com> wrote:
>> ...there are SHA1 and MD5 checksums next to the JAR artifact - I can
>> modify our download code to download one of them and use it for the
>> consistency of downloaded bits check....
>
> To be useful the digests need to come from a trusted source.
>
> For Apache releases, as described at
> http://www.apache.org/dev/release-publishing.html the digests, keys
> etc. must come from ASF hosts. This allows the release archives to be
> downloaded via channels that don't need to be trusted.
>
> Although we do trust Maven Central in general, if you're designing a
> new mechanism I suggest doing the same - distribute the digests from
> https://dist.apache.org/repos/dist/release/ (*) and the binaries from
> wherever.
>
> Note also that MD5 shouldn't be trusted anymore, see
> natmchugh.blogspot.com/2014/10/how-i-created-two-images-with-same-md5.html
> for example.
>
> -Bertrand
>
> (*) I think that's the correct place but best check with ASF infra
Re: Checksum was: Download binaries from Maven
Posted by Emilian Bold <em...@gmail.com>.
> To be useful the digests need to come from a trusted source.
Indeed we can trust the hash included in the source tree.
Downloading the binary and the hash from the Maven repo doesn't guarantee
it's the same file originally added nor that there is no MITM going on.
--emi
On Thu, Dec 8, 2016 at 11:35 AM, Bertrand Delacretaz <bdelacretaz@apache.org
> wrote:
> Hi,
>
> On Thu, Dec 8, 2016 at 10:14 AM, Jaroslav Tulach
> <ja...@oracle.com> wrote:
> > ...there are SHA1 and MD5 checksums next to the JAR artifact - I can
> > modify our download code to download one of them and use it for the
> > consistency of downloaded bits check....
>
> To be useful the digests need to come from a trusted source.
>
> For Apache releases, as described at
> http://www.apache.org/dev/release-publishing.html the digests, keys
> etc. must come from ASF hosts. This allows the release archives to be
> downloaded via channels that don't need to be trusted.
>
> Although we do trust Maven Central in general, if you're designing a
> new mechanism I suggest doing the same - distribute the digests from
> https://dist.apache.org/repos/dist/release/ (*) and the binaries from
> wherever.
>
> Note also that MD5 shouldn't be trusted anymore, see
> natmchugh.blogspot.com/2014/10/how-i-created-two-images-with-same-md5.html
> for example.
>
> -Bertrand
>
> (*) I think that's the correct place but best check with ASF infra
>
Re: Checksum was: Download binaries from Maven
Posted by Bertrand Delacretaz <bd...@apache.org>.
Hi,
On Thu, Dec 8, 2016 at 10:14 AM, Jaroslav Tulach
<ja...@oracle.com> wrote:
> ...there are SHA1 and MD5 checksums next to the JAR artifact - I can
> modify our download code to download one of them and use it for the
> consistency of downloaded bits check....
To be useful the digests need to come from a trusted source.
For Apache releases, as described at
http://www.apache.org/dev/release-publishing.html the digests, keys
etc. must come from ASF hosts. This allows the release archives to be
downloaded via channels that don't need to be trusted.
Although we do trust Maven Central in general, if you're designing a
new mechanism I suggest doing the same - distribute the digests from
https://dist.apache.org/repos/dist/release/ (*) and the binaries from
wherever.
Note also that MD5 shouldn't be trusted anymore, see
natmchugh.blogspot.com/2014/10/how-i-created-two-images-with-same-md5.html
for example.
-Bertrand
(*) I think that's the correct place but best check with ASF infra