You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Patrick Wendell (JIRA)" <ji...@apache.org> on 2014/10/24 21:10:34 UTC
[jira] [Commented] (SPARK-4079) Snappy bundled with Spark does not
work on older Linux distributions
[ https://issues.apache.org/jira/browse/SPARK-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183317#comment-14183317 ]
Patrick Wendell commented on SPARK-4079:
----------------------------------------
What about just catching the exception and throwing a better one that tells users to switch to use LZF? In my experience users are fine with this kind of thing if you just give them an actionable and simple step to work around it.
I think it's a bit extreme to declare that Spark doesn't support CentOS5 because of this - if people want to package Spark to work on CentOS5 they can just change the default compression codec.
Could be a good idea to ask Snappy to add older versions as well. The maintainer of that library is very responsive.
> Snappy bundled with Spark does not work on older Linux distributions
> --------------------------------------------------------------------
>
> Key: SPARK-4079
> URL: https://issues.apache.org/jira/browse/SPARK-4079
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.0.0
> Reporter: Marcelo Vanzin
>
> This issue has existed at least since 1.0, but has been made worse by 1.1 since snappy is now the default compression algorithm. When trying to use it on a CentOS 5 machine, for example, you'll get something like this:
> {noformat}
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:319)
> at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:226)
> at org.xerial.snappy.Snappy.<clinit>(Snappy.java:48)
> at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:79)
> at org.apache.spark.io.SnappyCompressionCodec.compressedOutputStream(CompressionCodec.scala:125)
> at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:207)
> ...
> Caused by: java.lang.UnsatisfiedLinkError: /tmp/snappy-1.0.5.3-af72bf3c-9dab-43af-a662-f9af657f06b1-libsnappyjava.so: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.9' not found (required by /tmp/snappy-1.0.5.3-af72bf3c-9dab-43af-a662-f9af657f06b1-libsnappyjava.so)
> at java.lang.ClassLoader$NativeLibrary.load(Native Method)
> at java.lang.ClassLoader.loadLibrary1(ClassLoader.java:1957)
> at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1882)
> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1843)
> at java.lang.Runtime.load0(Runtime.java:795)
> at java.lang.System.load(System.java:1061)
> at org.xerial.snappy.SnappyNativeLoader.load(SnappyNativeLoader.java:39)
> ... 29 more
> {noformat}
> There are two approaches I can see here (well, 3):
> * Declare CentOS 5 (and similar OSes) not supported, although that would suck for the people who are still on it and already use Spark
> * Fallback to another compression codec if Snappy cannot be loaded
> * Ask the Snappy guys to compile the library on an older OS...
> I think the second would be the best compromise.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org