[jira] [Created] (SPARK-4079) Snappy bundled with Spark does not work on older Linux distributions

["Marcelo Vanzin (JIRA)" <ji...@apache.org>]

Marcelo Vanzin created SPARK-4079:
-------------------------------------

             Summary: Snappy bundled with Spark does not work on older Linux distributions
                 Key: SPARK-4079
                 URL: https://issues.apache.org/jira/browse/SPARK-4079
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 1.0.0
            Reporter: Marcelo Vanzin


This issue has existed at least since 1.0, but has been made worse by 1.1 since snappy is now the default compression algorithm. When trying to use it on a CentOS 5 machine, for example, you'll get something like this:

{noformat}
      java.lang.reflect.InvocationTargetException
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
           at java.lang.reflect.Method.invoke(Method.java:606)
           at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:319)
           at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:226)
           at org.xerial.snappy.Snappy.<clinit>(Snappy.java:48)
           at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:79)
           at org.apache.spark.io.SnappyCompressionCodec.compressedOutputStream(CompressionCodec.scala:125)
           at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:207)
       ...
       Caused by: java.lang.UnsatisfiedLinkError: /tmp/snappy-1.0.5.3-af72bf3c-9dab-43af-a662-f9af657f06b1-libsnappyjava.so: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.9' not found (required by /tmp/snappy-1.0.5.3-af72bf3c-9dab-43af-a662-f9af657f06b1-libsnappyjava.so)
           at java.lang.ClassLoader$NativeLibrary.load(Native Method)
           at java.lang.ClassLoader.loadLibrary1(ClassLoader.java:1957)
           at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1882)
           at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1843)
           at java.lang.Runtime.load0(Runtime.java:795)
           at java.lang.System.load(System.java:1061)
           at org.xerial.snappy.SnappyNativeLoader.load(SnappyNativeLoader.java:39)
           ... 29 more
{noformat}

There are two approaches I can see here (well, 3):

* Declare CentOS 5 (and similar OSes) not supported, although that would suck for the people who are still on it and already use Spark
* Fallback to another compression codec if Snappy cannot be loaded
* Ask the Snappy guys to compile the library on an older OS...

I think the second would be the best compromise.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org