You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Marek Miglinski <mm...@seven.com> on 2012/06/11 15:54:59 UTC
snappy codec
Hi,
I have a Clouderas CDH3u3 installed on my cluster and mapred.child.env set to "LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64" (with libsnappy.so in the folder) in mapred-site.xml. Cloudera says that Snappy is included in their hadoop-0.20-native package and it is also installed on each of the nodes.
But when I run a mapreduce task with "mapred.map.output.compression.codec" set to "org.apache.hadoop.io.compress.SnappyCodec" I get an exception:
java.lang.RuntimeException: native snappy library not available
Any idea why?
Thanks,
Marek M.
Re: snappy codec
Posted by Harsh J <ha...@cloudera.com>.
Hi Marek,
Moving this to cdh-user@cloudera.org as its CDH specific. I've bcc'd
mapreduce-user@ and cc'd you in case you aren't a subscriber (@
https://groups.google.com/a/cloudera.org/group/cdh-user)
A few questions though:
- What OS are you running and what arch? Check via "lsb_release -a"
and "uname -a"
-- Is your JVM also same as your arch? Check via
"/usr/java/default/bin/java -version"
- Why do you provide
LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64 as
mapred.child.env? It isn't usually required if you have
hadoop-0.20-native installed and the TaskTrackers were restarted since
then.
- Do other codecs (such as GZip and Deflate (Default)) work fine?
On Mon, Jun 11, 2012 at 7:24 PM, Marek Miglinski <mm...@seven.com> wrote:
> Hi,
>
> I have a Clouderas CDH3u3 installed on my cluster and mapred.child.env set to "LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64" (with libsnappy.so in the folder) in mapred-site.xml. Cloudera says that Snappy is included in their hadoop-0.20-native package and it is also installed on each of the nodes.
>
> But when I run a mapreduce task with "mapred.map.output.compression.codec" set to "org.apache.hadoop.io.compress.SnappyCodec" I get an exception:
>
> java.lang.RuntimeException: native snappy library not available
>
>
> Any idea why?
>
>
>
> Thanks,
> Marek M.
--
Harsh J
Re: snappy codec
Posted by Mohit Anchlia <mo...@gmail.com>.
use java lib path
On Mon, Jun 11, 2012 at 7:12 AM, GUOJUN Zhu <gu...@freddiemac.com>wrote:
>
> I believe that the mapred.child.env only concerns the spawned child
> process where the map is run. You probably need to put this path into the
> system property where the map task demon runs and possibly where the job
> controller runs.
>
> Zhu, Guojun
> Modeling Sr Graduate
> 571-3824370
> guojun_zhu@freddiemac.com
> Financial Engineering
> Freddie Mac
>
>
> *Marek Miglinski <mm...@seven.com>*
>
> 06/11/2012 09:54 AM
> Please respond to
> mapreduce-user@hadoop.apache.org
>
> To
> "mapreduce-user@hadoop.apache.org" <ma...@hadoop.apache.org>
> cc
> Subject
> snappy codec
>
>
>
>
> Hi,
>
> I have a Clouderas CDH3u3 installed on my cluster and mapred.child.env set
> to "LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64" (with
> libsnappy.so in the folder) in mapred-site.xml. Cloudera says that Snappy
> is included in their hadoop-0.20-native package and it is also installed on
> each of the nodes.
>
> But when I run a mapreduce task with "mapred.map.output.compression.codec"
> set to "org.apache.hadoop.io.compress.SnappyCodec" I get an exception:
>
> java.lang.RuntimeException: native snappy library not available
>
>
> Any idea why?
>
>
>
> Thanks,
> Marek M.
>
>
Re: snappy codec
Posted by GUOJUN Zhu <gu...@freddiemac.com>.
I believe that the mapred.child.env only concerns the spawned child
process where the map is run. You probably need to put this path into the
system property where the map task demon runs and possibly where the job
controller runs.
Zhu, Guojun
Modeling Sr Graduate
571-3824370
guojun_zhu@freddiemac.com
Financial Engineering
Freddie Mac
Marek Miglinski <mm...@seven.com>
06/11/2012 09:54 AM
Please respond to
mapreduce-user@hadoop.apache.org
To
"mapreduce-user@hadoop.apache.org" <ma...@hadoop.apache.org>
cc
Subject
snappy codec
Hi,
I have a Clouderas CDH3u3 installed on my cluster and mapred.child.env set
to "LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64" (with
libsnappy.so in the folder) in mapred-site.xml. Cloudera says that Snappy
is included in their hadoop-0.20-native package and it is also installed
on each of the nodes.
But when I run a mapreduce task with "mapred.map.output.compression.codec"
set to "org.apache.hadoop.io.compress.SnappyCodec" I get an exception:
java.lang.RuntimeException: native snappy library not available
Any idea why?
Thanks,
Marek M.