Re: [crypto] Interest in adding support for cryptographic hash function?

[Alex Remily <al...@gmail.com>]

I echo all sentiments regarding benchmarking.  The deck at the link is a
bit dated, but it contains some benchmarking of the existing commons
crypto implementation against the JCE on JDK 8 and 9 that provides reason
for optimism.  Even if more recent versions of the JCE significantly narrow
the performance gap, which remains to be seen, I think commons crypto will
endure for the foreseeable future for users who need the performance and
can't or won't upgrade.

https://events.static.linuxfound.org/sites/events/files/slides/apache-commons-crypto-another-wheel.pdf

Regarding portability, the JNI libraries for each supported OS distribution
are bundled with the JAR and transparent to the caller of the commons
crypto api.  Yes, that makes it a hassle to build a release, but users need
only have libcrypto installed locally.



On Sun, Feb 28, 2021 at 6:21 PM Matt Sicker <bo...@gmail.com> wrote:

> That's why I'm interested in proper benchmarks before supporting a
> release of something with platform-specific code. The CPU extensions
> are ostensibly for optimizing the code where possible (the Blake3 code
> has a dynamic dispatch mechanism for checking CPU features at runtime
> to select the assembler-optimized variants), but if JNI overhead
> negates the gains there, then I'd agree that sticking to pure Java
> code here would be optimal.
>
> On Sun, 28 Feb 2021 at 17:18, sebb <se...@gmail.com> wrote:
> >
> > On Sun, 28 Feb 2021 at 20:14, Matt Sicker <bo...@gmail.com> wrote:
> > >
> > > I'd also be interested in benchmarking comparisons as I've been
> > > working on a proof of concept using Blake3 to do similarly (I have a
> > > pure Java implementation and a JNI version that ultimately invokes the
> > > reference C implementation, though I've also wondered about linking
> > > the reference Rust implementation). The potential advantage to linking
> > > in the native implementations here would be in taking advantage of
> > > CPU-specific extensions like SSE, AVX, NEON, etc., which aren't
> > > accessible from the JVM without JNI or other patches to the JDK
> > > itself. If the overhead turns out to be non-negligible, then we should
> > > probably keep the native code bindings to commons-crypto while porting
> > > pure Java implementations to commons-codec.
> >
> > Of course the disadvantage is that the code is not portable.
> > Also each OS version will need separate testing and makes releasing
> > binaries harder.
> > This is an issue we have already faced with Crypto.
> > And there will be versions of Java for which it will not work.
> >
> > So unless the is a great performance improvement, I don't think it's
> > worth the extra overheads.
> >
> > > On Sun, 28 Feb 2021 at 09:25, Bernd Eckenfels <ec...@zusammenkunft.net>
> wrote:
> > > >
> > > > ... and also do benchmarking early, the native interface overhead
> might be a problem so that pure java (intrinsic) code is much faster
> > > >
> > > >
> > > > --
> > > > http://bernd.eckenfels.net
> > > > ________________________________
> > > > Von: Gary Gregory <ga...@gmail.com>
> > > > Gesendet: Sunday, February 28, 2021 2:45:26 PM
> > > > An: Commons Developers List <de...@commons.apache.org>
> > > > Betreff: Re: [crypto] Interest in adding support for cryptographic
> hash function?
> > > >
> > > > That sounds reasonable to me. I think once we see a PR, we'll get a
> better
> > > > idea. Start small IMO.
> > > >
> > > > Gary
> > > >
> > > >
> > > > On Sat, Feb 27, 2021, 13:51 Alex Remily <al...@gmail.com>
> wrote:
> > > >
> > > > > I'd be exposing additional elements of the OpenSSL API via
> additions to the
> > > > > commons crypto API.  Since I wouldn't be adding any additional
> > > > > dependencies, I would expect that licensing and portability would
> remain
> > > > > unchanged.  Would it not?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Alex
> > > > >
> > > > > On Sat, Feb 27, 2021 at 1:08 PM Gilles Sadowski <
> gilleseran@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Le sam. 27 févr. 2021 à 19:00, Bernd Eckenfels
> > > > > > <ec...@zusammenkunft.net> a écrit :
> > > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > I don’t think it’s a Good idea to introduce native
> dependencies to
> > > > > > formerly pure Java projects.
> > > > > >
> > > > > > +1
> > > > > > [I thought that the idea was a (pure) Java implementation.]
> > > > > >
> > > > > > > So i think native optimized hash implementations would fit
> better in
> > > > > > commons-crypto. So I would say go for it, keep in mind license
> clearance
> > > > > > and portability.
> > > > > > >
> > > > > > > Gruß
> > > > > > > Bernd
> > > > > > >
> > > > > > > > [...]
> > > > > >
> > > > > >
> ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> > > > > > For additional commands, e-mail: dev-help@commons.apache.org
> > > > > >
> > > > > >
> > > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> > > For additional commands, e-mail: dev-help@commons.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> > For additional commands, e-mail: dev-help@commons.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
>

Re: [crypto] Interest in adding support for cryptographic hash function?

[Matt Sicker <bo...@gmail.com>]

Coming back to this thread since I started porting the Blake3 code.
Seeing the AES differences makes a lot of sense since most of the
software optimizations possible are insecure, and the only secure
software designs for AES have noticeable performance penalties
compared to AES intrinsics.

I'd still be interested to see what kind of improvements we can get
for other algorithms that can rely on CPU extensions similarly. I
recall seeing several on Intel's list [1] plus rdseed, etc..

[1]: https://software.intel.com/sites/landingpage/IntrinsicsGuide/#cats=Cryptography

On Sun, 28 Feb 2021 at 18:26, Alex Remily <al...@gmail.com> wrote:
>
> I echo all sentiments regarding benchmarking.  The deck at the link is a
> bit dated, but it contains some benchmarking of the existing commons
> crypto implementation against the JCE on JDK 8 and 9 that provides reason
> for optimism.  Even if more recent versions of the JCE significantly narrow
> the performance gap, which remains to be seen, I think commons crypto will
> endure for the foreseeable future for users who need the performance and
> can't or won't upgrade.
>
> https://events.static.linuxfound.org/sites/events/files/slides/apache-commons-crypto-another-wheel.pdf
>
> Regarding portability, the JNI libraries for each supported OS distribution
> are bundled with the JAR and transparent to the caller of the commons
> crypto api.  Yes, that makes it a hassle to build a release, but users need
> only have libcrypto installed locally.
>
>
>
> On Sun, Feb 28, 2021 at 6:21 PM Matt Sicker <bo...@gmail.com> wrote:
>
> > That's why I'm interested in proper benchmarks before supporting a
> > release of something with platform-specific code. The CPU extensions
> > are ostensibly for optimizing the code where possible (the Blake3 code
> > has a dynamic dispatch mechanism for checking CPU features at runtime
> > to select the assembler-optimized variants), but if JNI overhead
> > negates the gains there, then I'd agree that sticking to pure Java
> > code here would be optimal.
> >
> > On Sun, 28 Feb 2021 at 17:18, sebb <se...@gmail.com> wrote:
> > >
> > > On Sun, 28 Feb 2021 at 20:14, Matt Sicker <bo...@gmail.com> wrote:
> > > >
> > > > I'd also be interested in benchmarking comparisons as I've been
> > > > working on a proof of concept using Blake3 to do similarly (I have a
> > > > pure Java implementation and a JNI version that ultimately invokes the
> > > > reference C implementation, though I've also wondered about linking
> > > > the reference Rust implementation). The potential advantage to linking
> > > > in the native implementations here would be in taking advantage of
> > > > CPU-specific extensions like SSE, AVX, NEON, etc., which aren't
> > > > accessible from the JVM without JNI or other patches to the JDK
> > > > itself. If the overhead turns out to be non-negligible, then we should
> > > > probably keep the native code bindings to commons-crypto while porting
> > > > pure Java implementations to commons-codec.
> > >
> > > Of course the disadvantage is that the code is not portable.
> > > Also each OS version will need separate testing and makes releasing
> > > binaries harder.
> > > This is an issue we have already faced with Crypto.
> > > And there will be versions of Java for which it will not work.
> > >
> > > So unless the is a great performance improvement, I don't think it's
> > > worth the extra overheads.
> > >
> > > > On Sun, 28 Feb 2021 at 09:25, Bernd Eckenfels <ec...@zusammenkunft.net>
> > wrote:
> > > > >
> > > > > ... and also do benchmarking early, the native interface overhead
> > might be a problem so that pure java (intrinsic) code is much faster
> > > > >
> > > > >
> > > > > --
> > > > > http://bernd.eckenfels.net
> > > > > ________________________________
> > > > > Von: Gary Gregory <ga...@gmail.com>
> > > > > Gesendet: Sunday, February 28, 2021 2:45:26 PM
> > > > > An: Commons Developers List <de...@commons.apache.org>
> > > > > Betreff: Re: [crypto] Interest in adding support for cryptographic
> > hash function?
> > > > >
> > > > > That sounds reasonable to me. I think once we see a PR, we'll get a
> > better
> > > > > idea. Start small IMO.
> > > > >
> > > > > Gary
> > > > >
> > > > >
> > > > > On Sat, Feb 27, 2021, 13:51 Alex Remily <al...@gmail.com>
> > wrote:
> > > > >
> > > > > > I'd be exposing additional elements of the OpenSSL API via
> > additions to the
> > > > > > commons crypto API.  Since I wouldn't be adding any additional
> > > > > > dependencies, I would expect that licensing and portability would
> > remain
> > > > > > unchanged.  Would it not?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Alex
> > > > > >
> > > > > > On Sat, Feb 27, 2021 at 1:08 PM Gilles Sadowski <
> > gilleseran@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Le sam. 27 févr. 2021 à 19:00, Bernd Eckenfels
> > > > > > > <ec...@zusammenkunft.net> a écrit :
> > > > > > > >
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > I don’t think it’s a Good idea to introduce native
> > dependencies to
> > > > > > > formerly pure Java projects.
> > > > > > >
> > > > > > > +1
> > > > > > > [I thought that the idea was a (pure) Java implementation.]
> > > > > > >
> > > > > > > > So i think native optimized hash implementations would fit
> > better in
> > > > > > > commons-crypto. So I would say go for it, keep in mind license
> > clearance
> > > > > > > and portability.
> > > > > > > >
> > > > > > > > Gruß
> > > > > > > > Bernd
> > > > > > > >
> > > > > > > > > [...]
> > > > > > >
> > > > > > >
> > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> > > > > > > For additional commands, e-mail: dev-help@commons.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> > > > For additional commands, e-mail: dev-help@commons.apache.org
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> > > For additional commands, e-mail: dev-help@commons.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> > For additional commands, e-mail: dev-help@commons.apache.org
> >
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org