You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Jan Høydahl <ja...@cominvent.com> on 2017/03/01 15:13:48 UTC
Stop using SHA-1 and MD5 hashes?
Hi devs,
Working on LUCENE-5143 I’m revising the README.html files we place in the dist folders.
Then I started documenting how to validate checksum of the downloads in addition to GPG signature,
Looks like MD5 can still be used for integrity checks (https://en.wikipedia.org/wiki/MD5),
while the Ant guys claim otherwise in https://ant.apache.org/manual/Tasks/checksum.html
Will our .md5 and .sha1 files still provide security for the downloader after Google releases their
recent findings or are they only useful to check that the download was complete and not partial?
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Stop using SHA-1 and MD5 hashes?
Posted by Jan Høydahl <ja...@cominvent.com>.
See my proposed new REDME.html instructions from LUCENE-5143:
> Signatures and hashes
>
> <>All official source and binary releases are digitally signed using GnuPG. You are encouraged to verify that your download is the official one by verifying the digital signature. To do this you need, in addition to the downloaded file:
>
> the pgp or gpg software
> the official KEYS file for the project
> the file.asc file corresponding to your download
> Always download the KEYS and .asc files directly from the Apache site at <https://www.apache.org/dist/lucene/java/ <https://www.apache.org/dist/lucene/java/>>, and always over HTTPS. Never trust KEYS from a mirror site.
>
> Always test available signatures, e.g.,
> $ pgpk -a KEYS
> $ pgpk lucene-x.y.z.tar.gz.asc
> or,
> $ pgp -ka KEYS
> $ pgp lucene-x.y.z.tar.gz.asc
> or,
> $ gpg --import KEYS
> $ gpg --verify lucene-x.y.z.tar.gz.asc
> Checking the hashes
>
> Alongside the release artifacts in the official Apache dist site you will also find two other files providing checksum hashes for each file. E.g. for lucene-x.y.z.tgz there will be a lucene-x.y.z.tgz.sha1 file providing the SHA-1 checksum and a lucene-x.y.z.tgz.md5 file providing the MD5 checksum. These are useful to verify that your download was complete and valid, but will not prove that your download was digitally signed by an actual Apache committer. For that you must check the .asc signature.
>
> Calculate the checksum of your download and compare to the contents of the .sha1/.md5 files
> $ shasum lucene-x.y.z.tgz
> $ md5 lucene-x.y.z.tgz
Also noting that the md5 and sha1 files are not present in mirrors I recommend fetching them from apache dist site.
The subversion project does not copy asc or md5 to mirrors, but the hadoop project copies the .mds files...
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
> 2. mar. 2017 kl. 23.07 skrev Shawn Heisey <ap...@elyograg.org>:
>
> On 3/1/2017 8:13 AM, Jan Høydahl wrote:
>> Working on LUCENE-5143 I’m revising the README.html files we place in
>> the dist folders. Then I started documenting how to validate checksum
>> of the downloads in addition to GPG signature, Looks like MD5 can
>> still be used for integrity checks
>> (https://en.wikipedia.org/wiki/MD5), while the Ant guys claim
>> otherwise in https://ant.apache.org/manual/Tasks/checksum.html Will
>> our .md5 and .sha1 files still provide security for the downloader
>> after Google releases their recent findings or are they only useful to
>> check that the download was complete and not partial?
>
> From what I can see, hashes and signatures are both missing on the
> download mirrors for Lucene and Solr. That's probably prudent for
> hashes, but should signatures be there?
>
> I'd expect hashes to be used as a quick "did it download right?" check.
> It's a weak form of authentication also, but as researchers have found,
> definitely not foolproof. Also, any download location with an altered
> archive could have altered hashes.
>
> I do not think it would be possible for non-committers to create an
> altered GPG signature that validates, as long as the end user obtained
> the KEYS file directly from Apache. If I'm wrong about that, then
> perhaps we need an entirely new method of validation.
>
> Thanks,
> Shawn
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
Re: Stop using SHA-1 and MD5 hashes?
Posted by Jan Høydahl <ja...@cominvent.com>.
As per Apache policy the sigs or hashes should NOT be mirrored, see also
https://issues.apache.org/jira/browse/INFRA-6848 <https://issues.apache.org/jira/browse/INFRA-6848>
You can trust that it is an Apache committer who signed the artifact
by verifying with gpg —verify. If you got the KEYS file from Apache
and also double check the PGP key ID, then you are safe.
Ideally, all committers should get their code signing key signed by
the other committers to build a web of trust, but that is not done enough..
https://www.apache.org/dev/release-signing.html#web-of-trust <https://www.apache.org/dev/release-signing.html#web-of-trust>
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
> 3. mar. 2017 kl. 14.16 skrev Bram Van Dam <br...@intix.eu>:
>
>> From what I can see, hashes and signatures are both missing on the
>> download mirrors for Lucene and Solr. That's probably prudent for
>> hashes, but should signatures be there?
>
> I vaguely remembering raising this issue before -- though it might have
> been regarding a different Apache project. From what I remember, the ASF
> signature guidelines don't require software signing keys to be signed by
> anyone in particular. So unless the signature file is on the (https)
> Apache download site, it's probably effectively useless.
>
> After all there's nothing stopping me from setting up a rogue mirror,
> creating a "Shawn Heisey <ap...@elyograg.org>" GPG key and signing my
> fake release with it.
>
> Including signatures on mirrors would only lead to sloppy verification
> by whoever is downloading the software.
>
> That is, unless there's some kind of web of trust in the release
> signature, but that currently doesn't seem to be the case.
>
> - Bram
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
Re: Stop using SHA-1 and MD5 hashes?
Posted by Bram Van Dam <br...@intix.eu>.
> From what I can see, hashes and signatures are both missing on the
> download mirrors for Lucene and Solr. That's probably prudent for
> hashes, but should signatures be there?
I vaguely remembering raising this issue before -- though it might have
been regarding a different Apache project. From what I remember, the ASF
signature guidelines don't require software signing keys to be signed by
anyone in particular. So unless the signature file is on the (https)
Apache download site, it's probably effectively useless.
After all there's nothing stopping me from setting up a rogue mirror,
creating a "Shawn Heisey <ap...@elyograg.org>" GPG key and signing my
fake release with it.
Including signatures on mirrors would only lead to sloppy verification
by whoever is downloading the software.
That is, unless there's some kind of web of trust in the release
signature, but that currently doesn't seem to be the case.
- Bram
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Stop using SHA-1 and MD5 hashes?
Posted by Shawn Heisey <ap...@elyograg.org>.
On 3/1/2017 8:13 AM, Jan H�ydahl wrote:
> Working on LUCENE-5143 I\u2019m revising the README.html files we place in
> the dist folders. Then I started documenting how to validate checksum
> of the downloads in addition to GPG signature, Looks like MD5 can
> still be used for integrity checks
> (https://en.wikipedia.org/wiki/MD5), while the Ant guys claim
> otherwise in https://ant.apache.org/manual/Tasks/checksum.html Will
> our .md5 and .sha1 files still provide security for the downloader
> after Google releases their recent findings or are they only useful to
> check that the download was complete and not partial?
From what I can see, hashes and signatures are both missing on the
download mirrors for Lucene and Solr. That's probably prudent for
hashes, but should signatures be there?
I'd expect hashes to be used as a quick "did it download right?" check.
It's a weak form of authentication also, but as researchers have found,
definitely not foolproof. Also, any download location with an altered
archive could have altered hashes.
I do not think it would be possible for non-committers to create an
altered GPG signature that validates, as long as the end user obtained
the KEYS file directly from Apache. If I'm wrong about that, then
perhaps we need an entirely new method of validation.
Thanks,
Shawn
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org