[jira] [Assigned] (SPARK-10949) Upgrade Snappy Java to 1.1.2

["Apache Spark (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-10949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-10949:
------------------------------------

    Assignee: Apache Spark

> Upgrade Snappy Java to 1.1.2
> ----------------------------
>
>                 Key: SPARK-10949
>                 URL: https://issues.apache.org/jira/browse/SPARK-10949
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 1.5.0, 1.5.1
>            Reporter: Adam Roberts
>            Assignee: Apache Spark
>            Priority: Minor
>
> Snappy now supports concatenation of serialized streams, this patch contains a version number change and the "does not support" test is now a "supports" test.
> Note: I do have the pull request for this already created, tested to be OK on Intel 64 bit Linux, IBM Power 8 LE and Linux on IBM Z Systems (all with IBM Java 8). Also note that I was required to delete my m2 cache in order to resolve incompatible class exceptions when building.
> Snappy 1.1.2 changelog mentions:
> snappy-java-1.1.2 (22 September 2015)
> This is a backward compatible release for 1.1.x.
> Add AIX (32-bit) support.
> There is no upgrade for the native libraries of the other platforms.
> A major change since 1.1.1 is a support for reading concatenated results of SnappyOutputStream(s)
> snappy-java-1.1.2-RC2 (18 May 2015)
> Fix #107: SnappyOutputStream.close() is not idempotent
> snappy-java-1.1.2-RC1 (13 May 2015)
> SnappyInputStream now supports reading concatenated compressed results of SnappyOutputStream
> There has been no compressed format change since 1.0.5.x. So You can read the compressed results interchangeablly between these versions.
> Fixes a problem when java.io.tmpdir does not exist.
> From https://github.com/xerial/snappy-java/blob/develop/Milestone.md and up to date at the time of this pull request
> Also note https://github.com/xerial/snappy-java/issues/103
> "@xerial not sure how feasible or likely it is for this to happen, but it'd help tremendously Spark's performance because we are experimenting with a new shuffle path that uses channel.transferTo to avoid user space copying. However, for that to work, we'd need the underlying format to support concatenation. As far we know, LZF has this property, and Snappy might also have it (but snappy-java implementation doesn't support it)."



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org