Snappy compressor (native->java)

[Sai Boorlagadda <sa...@apache.org>]

Geode currently bundles xerial/snappy
<https://github.com/xerial/snappy-java> as a default implementation. And
this is a "JNI wrapper" on google snappy <http://google.github.io/snappy/>
implementation.

"xerial/snappy" jar bundles several pre-compiled static libraries to
support various OS (linux, windows, SunOS) and architectures (x86, Sparc
etc). The current dependency (1.1.1.6) does not support SunOS (Sparc), so
the plan is to upgrade to a more recent version.

While upgrading to a more recent version, I found a pure java port
<https://github.com/dain/snappy> of original C++ implementation and wanted
to swap with use pure java implementation to avoid any platform specific
dependency on Geode.

From the creator - "*the pure Java port is 20-30% faster for block
compress, 0-10% slower for block uncompress, and 0-5% slower for round-trip
block compression.*".

Though native version is better on uncompress (more number of gets, puts
depending on use cases), I would still vote for distributing with a pure
java version as a "default" implementation as Geode already exposes an
interface to allow any one to provide any custom implementation.

In case if there are any differences between these two implementations,
swapping with a pure java version should not impact any existing users, as
Geode does not save compressed data to disk or on to the wire.

Let me know if any one has different thoughts?

Sai

Re: Snappy compressor (native->java)

[Darrel Schneider <ds...@pivotal.io>]

+1
A problem with the native bits is this product feature only works on
certain native platforms.


On Mon, Jun 20, 2016 at 7:48 PM, Sai Boorlagadda <sa...@gmail.com>
wrote:

> There were no problems on the JNI wrapper (with native bits) except that
> maintaining native bits needs special development & testing efforts than
> pure java one.
>
> On Mon, Jun 20, 2016 at 5:52 PM, Dan Smith <ds...@pivotal.io> wrote:
>
> > +0  If the native code is causing problems switching seems like a good
> > idea.
> >
> > I would suggest looking into embedding the native bits into a JAR file.
> > > There are some tricks you can do to write the native bits out to disk
> > from
> > > the JAR file to load them at runtime. This would make it easier for
> > someone
> > > to deploy the speedup by just dropping in a single JAR into the class
> > path.
> > >
> >
> > That's what the xerial snappy jar does, actually. The native bits are all
> > in the jar file.
> >
> > -Dan
> >
>

Re: Snappy compressor (native->java)

[Sai Boorlagadda <sa...@gmail.com>]

There were no problems on the JNI wrapper (with native bits) except that
maintaining native bits needs special development & testing efforts than
pure java one.

On Mon, Jun 20, 2016 at 5:52 PM, Dan Smith <ds...@pivotal.io> wrote:

> +0  If the native code is causing problems switching seems like a good
> idea.
>
> I would suggest looking into embedding the native bits into a JAR file.
> > There are some tricks you can do to write the native bits out to disk
> from
> > the JAR file to load them at runtime. This would make it easier for
> someone
> > to deploy the speedup by just dropping in a single JAR into the class
> path.
> >
>
> That's what the xerial snappy jar does, actually. The native bits are all
> in the jar file.
>
> -Dan
>

Re: Snappy compressor (native->java)

[Dan Smith <ds...@pivotal.io>]

+0  If the native code is causing problems switching seems like a good idea.

I would suggest looking into embedding the native bits into a JAR file.
> There are some tricks you can do to write the native bits out to disk from
> the JAR file to load them at runtime. This would make it easier for someone
> to deploy the speedup by just dropping in a single JAR into the class path.
>

That's what the xerial snappy jar does, actually. The native bits are all
in the jar file.

-Dan

Re: Snappy compressor (native->java)

[Michael Stolz <ms...@pivotal.io>]

+1 pure java

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771
On Jun 20, 2016 4:58 PM, "Jacob Barrett" <jb...@pivotal.io> wrote:

> +1 for pure java default and making the native a drop in option.
>
> I would suggest looking into embedding the native bits into a JAR file.
> There are some tricks you can do to write the native bits out to disk from
> the JAR file to load them at runtime. This would make it easier for someone
> to deploy the speedup by just dropping in a single JAR into the class path.
>
> -Jake
>
>
> On Mon, Jun 20, 2016 at 4:14 PM Sai Boorlagadda <
> sai_boorlagadda@apache.org>
> wrote:
>
> > Geode currently bundles xerial/snappy
> > <https://github.com/xerial/snappy-java> as a default implementation. And
> > this is a "JNI wrapper" on google snappy <
> http://google.github.io/snappy/>
> > implementation.
> >
> > "xerial/snappy" jar bundles several pre-compiled static libraries to
> > support various OS (linux, windows, SunOS) and architectures (x86, Sparc
> > etc). The current dependency (1.1.1.6) does not support SunOS (Sparc), so
> > the plan is to upgrade to a more recent version.
> >
> > While upgrading to a more recent version, I found a pure java port
> > <https://github.com/dain/snappy> of original C++ implementation and
> wanted
> > to swap with use pure java implementation to avoid any platform specific
> > dependency on Geode.
> >
> > From the creator - "*the pure Java port is 20-30% faster for block
> > compress, 0-10% slower for block uncompress, and 0-5% slower for
> round-trip
> > block compression.*".
> >
> > Though native version is better on uncompress (more number of gets, puts
> > depending on use cases), I would still vote for distributing with a pure
> > java version as a "default" implementation as Geode already exposes an
> > interface to allow any one to provide any custom implementation.
> >
> > In case if there are any differences between these two implementations,
> > swapping with a pure java version should not impact any existing users,
> as
> > Geode does not save compressed data to disk or on to the wire.
> >
> > Let me know if any one has different thoughts?
> >
> > Sai
> >
>

Re: Snappy compressor (native->java)

[Jacob Barrett <jb...@pivotal.io>]

+1 for pure java default and making the native a drop in option.

I would suggest looking into embedding the native bits into a JAR file.
There are some tricks you can do to write the native bits out to disk from
the JAR file to load them at runtime. This would make it easier for someone
to deploy the speedup by just dropping in a single JAR into the class path.

-Jake


On Mon, Jun 20, 2016 at 4:14 PM Sai Boorlagadda <sa...@apache.org>
wrote:

> Geode currently bundles xerial/snappy
> <https://github.com/xerial/snappy-java> as a default implementation. And
> this is a "JNI wrapper" on google snappy <http://google.github.io/snappy/>
> implementation.
>
> "xerial/snappy" jar bundles several pre-compiled static libraries to
> support various OS (linux, windows, SunOS) and architectures (x86, Sparc
> etc). The current dependency (1.1.1.6) does not support SunOS (Sparc), so
> the plan is to upgrade to a more recent version.
>
> While upgrading to a more recent version, I found a pure java port
> <https://github.com/dain/snappy> of original C++ implementation and wanted
> to swap with use pure java implementation to avoid any platform specific
> dependency on Geode.
>
> From the creator - "*the pure Java port is 20-30% faster for block
> compress, 0-10% slower for block uncompress, and 0-5% slower for round-trip
> block compression.*".
>
> Though native version is better on uncompress (more number of gets, puts
> depending on use cases), I would still vote for distributing with a pure
> java version as a "default" implementation as Geode already exposes an
> interface to allow any one to provide any custom implementation.
>
> In case if there are any differences between these two implementations,
> swapping with a pure java version should not impact any existing users, as
> Geode does not save compressed data to disk or on to the wire.
>
> Let me know if any one has different thoughts?
>
> Sai
>