[jira] [Created] (HBASE-16553) Spark tests fail on some non Linux systems

["Andrew Purtell (JIRA)" <ji...@apache.org>]

Andrew Purtell created HBASE-16553:
--------------------------------------

             Summary: Spark tests fail on some non Linux systems 
                 Key: HBASE-16553
                 URL: https://issues.apache.org/jira/browse/HBASE-16553
             Project: HBase
          Issue Type: Bug
            Reporter: Andrew Purtell


The spark tests fail for me on FreeBSD. 

{noformat}
- bulkput to test HBase client *** FAILED ***
  java.lang.reflect.InvocationTargetException:
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
  at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
  at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
  at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
  at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
  at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
  at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
  ...
  Cause: java.lang.IllegalArgumentException: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=FreeBSD and os.arch=x86_64
  at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:156)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
  at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
  at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
  at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
  at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
  at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
  ...
  Cause: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=FreeBSD and os.arch=x86_64
  at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:331)
  at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:171)
  at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:152)
  at org.xerial.snappy.Snappy.<clinit>(Snappy.java:46)
  at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:154)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
  at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)

{noformat}

Maybe there's a configuration option to turn off snappy compression.

Either xerial snappy is trying to find an embedded library as a resource or the native library loading path isn't considering /usr/local/lib:
{noformat}
$ ls /usr/local/lib/*snappy*
/usr/local/lib/libsnappy.a		/usr/local/lib/libsnappy.so.1
/usr/local/lib/libsnappy.so		/usr/local/lib/libsnappy.so.1.3.0
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)