[jira] [Created] (SPARK-10949) Upgrade Snappy Java to 1.1.2

["Adam Roberts (JIRA)" <ji...@apache.org>]

Adam Roberts created SPARK-10949:
------------------------------------

             Summary: Upgrade Snappy Java to 1.1.2
                 Key: SPARK-10949
                 URL: https://issues.apache.org/jira/browse/SPARK-10949
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 1.5.1, 1.5.0, 1.5.2, 1.6.0
            Reporter: Adam Roberts
            Priority: Minor


Snappy now supports concatenation of serialized streams, this patch contains a version number change and the "does not support" test is now a "supports" test.

Note: I do have the pull request for this already created, tested to be OK on Intel 64 bit Linux, IBM Power 8 LE and Linux on IBM Z Systems (all with IBM Java 8). Also note that I was required to delete my m2 cache in order to resolve incompatible class exceptions when building.

Snappy 1.1.2 changelog mentions:

snappy-java-1.1.2 (22 September 2015)

This is a backward compatible release for 1.1.x.
Add AIX (32-bit) support.
There is no upgrade for the native libraries of the other platforms.
A major change since 1.1.1 is a support for reading concatenated results of SnappyOutputStream(s)
snappy-java-1.1.2-RC2 (18 May 2015)

Fix #107: SnappyOutputStream.close() is not idempotent
snappy-java-1.1.2-RC1 (13 May 2015)

SnappyInputStream now supports reading concatenated compressed results of SnappyOutputStream
There has been no compressed format change since 1.0.5.x. So You can read the compressed results interchangeablly between these versions.
Fixes a problem when java.io.tmpdir does not exist.

>From https://github.com/xerial/snappy-java/blob/develop/Milestone.md and up to date at the time of this pull request

Also note https://github.com/xerial/snappy-java/issues/103

"@xerial not sure how feasible or likely it is for this to happen, but it'd help tremendously Spark's performance because we are experimenting with a new shuffle path that uses channel.transferTo to avoid user space copying. However, for that to work, we'd need the underlying format to support concatenation. As far we know, LZF has this property, and Snappy might also have it (but snappy-java implementation doesn't support it)."



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org