You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Nick Travers <n....@gmail.com> on 2015/04/02 04:13:42 UTC
Spark, snappy and HDFS
Has anyone else encountered the following error when trying to read a snappy
compressed sequence file from HDFS?
*java.lang.UnsatisfiedLinkError:
org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z*
The following works for me when the file is uncompressed:
import org.apache.hadoop.io._
val hdfsPath = "hdfs://nost.name/path/to/folder"
val file = sc.sequenceFile[BytesWritable,String](hdfsPath)
file.count()
but fails when the encoding is Snappy.
I've seen some stuff floating around on the web about having to explicitly
enable support for Snappy in spark, but it doesn't seem to work for me:
http://www.ericlin.me/enabling-snappy-support-for-sharkspark
<http://www.ericlin.me/enabling-snappy-support-for-sharkspark>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-snappy-and-HDFS-tp22349.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark, snappy and HDFS
Posted by Nick Travers <n....@gmail.com>.
Thanks all. I was able to get the decompression working by adding the
following to my spark-env.sh script:
export
JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:/home/nickt/lib/hadoop/lib/native
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/nickt/lib/hadoop/lib/native
export
SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:/home/nickt/lib/hadoop/lib/native
export
SPARK_CLASSPATH=$SPARK_CLASSPATH:/home/nickt/lib/hadoop/lib/snappy-java-1.0.4.1.jar
On Thu, Apr 2, 2015 at 12:51 AM, Sean Owen <so...@cloudera.com> wrote:
> Yes, any Hadoop-related process that asks for Snappy compression or
> needs to read it will have to have the Snappy libs available on the
> library path. That's usually set up for you in a distro or you can do
> it manually like this. This is not Spark-specific.
>
> The second question also isn't Spark-specific; you do not have a
> SequenceFile of byte[] / String, but of byte[] / byte[]. Review what
> you are writing since it is not BytesWritable / Text.
>
> On Thu, Apr 2, 2015 at 3:40 AM, Nick Travers <n....@gmail.com>
> wrote:
> > I'm actually running this in a separate environment to our HDFS cluster.
> >
> > I think I've been able to sort out the issue by copying
> > /opt/cloudera/parcels/CDH/lib to the machine I'm running this on (I'm
> just
> > using a one-worker setup at present) and adding the following to
> > spark-env.sh:
> >
> > export
> > JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:/home/nickt/lib/hadoop/lib/native
> > export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/nickt/lib/hadoop/lib/native
> > export
> > SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:/home/nickt/lib/hadoop/lib/native
> > export
> >
> SPARK_CLASSPATH=$SPARK_CLASSPATH:/home/nickt/lib/hadoop/lib/snappy-java-1.0.4.1.jar
> >
> > I can get past the previous error. The issue now seems to be with what is
> > being returned.
> >
> > import org.apache.hadoop.io._
> > val hdfsPath = "hdfs://nost.name/path/to/folder"
> > val file = sc.sequenceFile[BytesWritable,String](hdfsPath)
> > file.count()
> >
> > returns the following error:
> >
> > java.lang.ClassCastException: org.apache.hadoop.io.BytesWritable cannot
> be
> > cast to org.apache.hadoop.io.Text
> >
> >
> > On Wed, Apr 1, 2015 at 7:34 PM, Xianjin YE <ad...@gmail.com> wrote:
> >>
> >> Do you have the same hadoop config for all nodes in your cluster(you run
> >> it in a cluster, right?)?
> >> Check the node(usually the executor) which gives the
> >> java.lang.UnsatisfiedLinkError to see whether the libsnappy.so is in the
> >> hadoop native lib path.
> >>
> >> On Thursday, April 2, 2015 at 10:22 AM, Nick Travers wrote:
> >>
> >> Thanks for the super quick response!
> >>
> >> I can read the file just fine in hadoop, it's just when I point Spark at
> >> this file it can't seem to read it due to the missing snappy jars /
> so's.
> >>
> >> I'l paying around with adding some things to spark-env.sh file, but
> still
> >> nothing.
> >>
> >> On Wed, Apr 1, 2015 at 7:19 PM, Xianjin YE <ad...@gmail.com>
> wrote:
> >>
> >> Can you read snappy compressed file in hdfs? Looks like the
> libsnappy.so
> >> is not in the hadoop native lib path.
> >>
> >> On Thursday, April 2, 2015 at 10:13 AM, Nick Travers wrote:
> >>
> >> Has anyone else encountered the following error when trying to read a
> >> snappy
> >> compressed sequence file from HDFS?
> >>
> >> *java.lang.UnsatisfiedLinkError:
> >> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z*
> >>
> >> The following works for me when the file is uncompressed:
> >>
> >> import org.apache.hadoop.io._
> >> val hdfsPath = "hdfs://nost.name/path/to/folder"
> >> val file = sc.sequenceFile[BytesWritable,String](hdfsPath)
> >> file.count()
> >>
> >> but fails when the encoding is Snappy.
> >>
> >> I've seen some stuff floating around on the web about having to
> explicitly
> >> enable support for Snappy in spark, but it doesn't seem to work for me:
> >> http://www.ericlin.me/enabling-snappy-support-for-sharkspark
> >> <http://www.ericlin.me/enabling-snappy-support-for-sharkspark>
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-snappy-and-HDFS-tp22349.html
> >> Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> >> For additional commands, e-mail: user-help@spark.apache.org
> >>
> >>
> >>
> >>
> >
>
Re: Spark, snappy and HDFS
Posted by Sean Owen <so...@cloudera.com>.
Yes, any Hadoop-related process that asks for Snappy compression or
needs to read it will have to have the Snappy libs available on the
library path. That's usually set up for you in a distro or you can do
it manually like this. This is not Spark-specific.
The second question also isn't Spark-specific; you do not have a
SequenceFile of byte[] / String, but of byte[] / byte[]. Review what
you are writing since it is not BytesWritable / Text.
On Thu, Apr 2, 2015 at 3:40 AM, Nick Travers <n....@gmail.com> wrote:
> I'm actually running this in a separate environment to our HDFS cluster.
>
> I think I've been able to sort out the issue by copying
> /opt/cloudera/parcels/CDH/lib to the machine I'm running this on (I'm just
> using a one-worker setup at present) and adding the following to
> spark-env.sh:
>
> export
> JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:/home/nickt/lib/hadoop/lib/native
> export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/nickt/lib/hadoop/lib/native
> export
> SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:/home/nickt/lib/hadoop/lib/native
> export
> SPARK_CLASSPATH=$SPARK_CLASSPATH:/home/nickt/lib/hadoop/lib/snappy-java-1.0.4.1.jar
>
> I can get past the previous error. The issue now seems to be with what is
> being returned.
>
> import org.apache.hadoop.io._
> val hdfsPath = "hdfs://nost.name/path/to/folder"
> val file = sc.sequenceFile[BytesWritable,String](hdfsPath)
> file.count()
>
> returns the following error:
>
> java.lang.ClassCastException: org.apache.hadoop.io.BytesWritable cannot be
> cast to org.apache.hadoop.io.Text
>
>
> On Wed, Apr 1, 2015 at 7:34 PM, Xianjin YE <ad...@gmail.com> wrote:
>>
>> Do you have the same hadoop config for all nodes in your cluster(you run
>> it in a cluster, right?)?
>> Check the node(usually the executor) which gives the
>> java.lang.UnsatisfiedLinkError to see whether the libsnappy.so is in the
>> hadoop native lib path.
>>
>> On Thursday, April 2, 2015 at 10:22 AM, Nick Travers wrote:
>>
>> Thanks for the super quick response!
>>
>> I can read the file just fine in hadoop, it's just when I point Spark at
>> this file it can't seem to read it due to the missing snappy jars / so's.
>>
>> I'l paying around with adding some things to spark-env.sh file, but still
>> nothing.
>>
>> On Wed, Apr 1, 2015 at 7:19 PM, Xianjin YE <ad...@gmail.com> wrote:
>>
>> Can you read snappy compressed file in hdfs? Looks like the libsnappy.so
>> is not in the hadoop native lib path.
>>
>> On Thursday, April 2, 2015 at 10:13 AM, Nick Travers wrote:
>>
>> Has anyone else encountered the following error when trying to read a
>> snappy
>> compressed sequence file from HDFS?
>>
>> *java.lang.UnsatisfiedLinkError:
>> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z*
>>
>> The following works for me when the file is uncompressed:
>>
>> import org.apache.hadoop.io._
>> val hdfsPath = "hdfs://nost.name/path/to/folder"
>> val file = sc.sequenceFile[BytesWritable,String](hdfsPath)
>> file.count()
>>
>> but fails when the encoding is Snappy.
>>
>> I've seen some stuff floating around on the web about having to explicitly
>> enable support for Snappy in spark, but it doesn't seem to work for me:
>> http://www.ericlin.me/enabling-snappy-support-for-sharkspark
>> <http://www.ericlin.me/enabling-snappy-support-for-sharkspark>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-snappy-and-HDFS-tp22349.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark, snappy and HDFS
Posted by Nick Travers <n....@gmail.com>.
I'm actually running this in a separate environment to our HDFS cluster.
I think I've been able to sort out the issue by copying
/opt/cloudera/parcels/CDH/lib to the machine I'm running this on (I'm just
using a one-worker setup at present) and adding the following to
spark-env.sh:
export
JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:/home/nickt/lib/hadoop/lib/native
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/nickt/lib/hadoop/lib/native
export
SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:/home/nickt/lib/hadoop/lib/native
export
SPARK_CLASSPATH=$SPARK_CLASSPATH:/home/nickt/lib/hadoop/lib/snappy-java-1.0.4.1.jar
I can get past the previous error. The issue now seems to be with what is
being returned.
import org.apache.hadoop.io._
val hdfsPath = "hdfs://nost.name/path/to/folder"
val file = sc.sequenceFile[BytesWritable,String](hdfsPath)
file.count()
returns the following error:
java.lang.ClassCastException: org.apache.hadoop.io.BytesWritable cannot be
cast to org.apache.hadoop.io.Text
On Wed, Apr 1, 2015 at 7:34 PM, Xianjin YE <ad...@gmail.com> wrote:
> Do you have the same hadoop config for all nodes in your cluster(you run
> it in a cluster, right?)?
> Check the node(usually the executor) which gives the
> java.lang.UnsatisfiedLinkError to see whether the libsnappy.so is in the
> hadoop native lib path.
>
> On Thursday, April 2, 2015 at 10:22 AM, Nick Travers wrote:
>
> Thanks for the super quick response!
>
> I can read the file just fine in hadoop, it's just when I point Spark at
> this file it can't seem to read it due to the missing snappy jars / so's.
>
> I'l paying around with adding some things to spark-env.sh file, but still
> nothing.
>
> On Wed, Apr 1, 2015 at 7:19 PM, Xianjin YE <ad...@gmail.com> wrote:
>
> Can you read snappy compressed file in hdfs? Looks like the libsnappy.so
> is not in the hadoop native lib path.
>
> On Thursday, April 2, 2015 at 10:13 AM, Nick Travers wrote:
>
> Has anyone else encountered the following error when trying to read a
> snappy
> compressed sequence file from HDFS?
>
> *java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z*
>
> The following works for me when the file is uncompressed:
>
> import org.apache.hadoop.io._
> val hdfsPath = "hdfs://nost.name/path/to/folder"
> val file = sc.sequenceFile[BytesWritable,String](hdfsPath)
> file.count()
>
> but fails when the encoding is Snappy.
>
> I've seen some stuff floating around on the web about having to explicitly
> enable support for Snappy in spark, but it doesn't seem to work for me:
> http://www.ericlin.me/enabling-snappy-support-for-sharkspark
> <http://www.ericlin.me/enabling-snappy-support-for-sharkspark>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-snappy-and-HDFS-tp22349.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
>
>
>
Re: Spark, snappy and HDFS
Posted by Xianjin YE <ad...@gmail.com>.
Do you have the same hadoop config for all nodes in your cluster(you run it in a cluster, right?)?
Check the node(usually the executor) which gives the java.lang.UnsatisfiedLinkError to see whether the libsnappy.so is in the hadoop native lib path.
On Thursday, April 2, 2015 at 10:22 AM, Nick Travers wrote:
> Thanks for the super quick response!
>
> I can read the file just fine in hadoop, it's just when I point Spark at this file it can't seem to read it due to the missing snappy jars / so's.
>
> I'l paying around with adding some things to spark-env.sh (http://spark-env.sh) file, but still nothing.
>
> On Wed, Apr 1, 2015 at 7:19 PM, Xianjin YE <advancedxy@gmail.com (mailto:advancedxy@gmail.com)> wrote:
> > Can you read snappy compressed file in hdfs? Looks like the libsnappy.so is not in the hadoop native lib path.
> >
> > On Thursday, April 2, 2015 at 10:13 AM, Nick Travers wrote:
> >
> > > Has anyone else encountered the following error when trying to read a snappy
> > > compressed sequence file from HDFS?
> > >
> > > *java.lang.UnsatisfiedLinkError:
> > > org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z*
> > >
> > > The following works for me when the file is uncompressed:
> > >
> > > import org.apache.hadoop.io._
> > > val hdfsPath = "hdfs://nost.name/path/to/folder (http://nost.name/path/to/folder)"
> > > val file = sc.sequenceFile[BytesWritable,String](hdfsPath)
> > > file.count()
> > >
> > > but fails when the encoding is Snappy.
> > >
> > > I've seen some stuff floating around on the web about having to explicitly
> > > enable support for Snappy in spark, but it doesn't seem to work for me:
> > > http://www.ericlin.me/enabling-snappy-support-for-sharkspark
> > > <http://www.ericlin.me/enabling-snappy-support-for-sharkspark>
> > >
> > >
> > >
> > > --
> > > View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-snappy-and-HDFS-tp22349.html
> > > Sent from the Apache Spark User List mailing list archive at Nabble.com (http://Nabble.com).
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org (mailto:user-unsubscribe@spark.apache.org)
> > > For additional commands, e-mail: user-help@spark.apache.org (mailto:user-help@spark.apache.org)
> > >
> > >
> > >
> >
> >
>
Re: Spark, snappy and HDFS
Posted by Nick Travers <n....@gmail.com>.
Thanks for the super quick response!
I can read the file just fine in hadoop, it's just when I point Spark at
this file it can't seem to read it due to the missing snappy jars / so's.
I'l paying around with adding some things to spark-env.sh file, but still
nothing.
On Wed, Apr 1, 2015 at 7:19 PM, Xianjin YE <ad...@gmail.com> wrote:
> Can you read snappy compressed file in hdfs? Looks like the libsnappy.so
> is not in the hadoop native lib path.
>
> On Thursday, April 2, 2015 at 10:13 AM, Nick Travers wrote:
>
> Has anyone else encountered the following error when trying to read a
> snappy
> compressed sequence file from HDFS?
>
> *java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z*
>
> The following works for me when the file is uncompressed:
>
> import org.apache.hadoop.io._
> val hdfsPath = "hdfs://nost.name/path/to/folder"
> val file = sc.sequenceFile[BytesWritable,String](hdfsPath)
> file.count()
>
> but fails when the encoding is Snappy.
>
> I've seen some stuff floating around on the web about having to explicitly
> enable support for Snappy in spark, but it doesn't seem to work for me:
> http://www.ericlin.me/enabling-snappy-support-for-sharkspark
> <http://www.ericlin.me/enabling-snappy-support-for-sharkspark>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-snappy-and-HDFS-tp22349.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
>
Re: Spark, snappy and HDFS
Posted by Xianjin YE <ad...@gmail.com>.
Can you read snappy compressed file in hdfs? Looks like the libsnappy.so is not in the hadoop native lib path.
On Thursday, April 2, 2015 at 10:13 AM, Nick Travers wrote:
> Has anyone else encountered the following error when trying to read a snappy
> compressed sequence file from HDFS?
>
> *java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z*
>
> The following works for me when the file is uncompressed:
>
> import org.apache.hadoop.io._
> val hdfsPath = "hdfs://nost.name/path/to/folder (http://nost.name/path/to/folder)"
> val file = sc.sequenceFile[BytesWritable,String](hdfsPath)
> file.count()
>
> but fails when the encoding is Snappy.
>
> I've seen some stuff floating around on the web about having to explicitly
> enable support for Snappy in spark, but it doesn't seem to work for me:
> http://www.ericlin.me/enabling-snappy-support-for-sharkspark
> <http://www.ericlin.me/enabling-snappy-support-for-sharkspark>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-snappy-and-HDFS-tp22349.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com (http://Nabble.com).
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org (mailto:user-unsubscribe@spark.apache.org)
> For additional commands, e-mail: user-help@spark.apache.org (mailto:user-help@spark.apache.org)
>
>