You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by Subroto Sanyal <sa...@gmail.com> on 2014/11/17 11:07:20 UTC
Native Compression Lib loading failed
Hi,
With HDP-2.1.2 we are encountering the problem of:
2014-09-11 10:14:42,881 ERROR [TezChild]
org.apache.tez.runtime.task.TezTaskRunner: Exception of type Error.
Exiting now
java.lang.UnsatisfiedLinkError:
org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native
Method)
at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132)
at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1198)
at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1091)
at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1441)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:274)
To workaround the problem we have to set the properties explicitly:
tez.am.launch.env
tez.task.launch.env
Is this a problem with HDP installation?
--
Cheers,
*Subroto Sanyal*
RE: Native Compression Lib loading failed
Posted by Bikas Saha <bi...@hortonworks.com>.
By default, native libs are expected at $HADOOP_COMMON_HOME/lib/native/ for
linux and added to the AM/task launch env via
"LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_COMMON_HOME/lib/native/". For
Windows the PATH is updated with the location instead of LD_LIBRARY_PATH.
If your native libs are in a different location, then the AM/task env would
need to be overridden with the location of the native lib.
*From:* Siddharth Seth [mailto:sseth@apache.org]
*Sent:* Monday, November 17, 2014 12:15 PM
*To:* user@tez.apache.org
*Subject:* Re: Native Compression Lib loading failed
Are these direct Tez jobs ?
The task and am env do need to be setup to make use of Snappy. The property
that needs to be set depends on the job being run. Tez direct would require
tez.am.launch.env and tez.task.launch.env to be set. For Hive, Pig, MR, etc
this may be different, depending on how these projects setup the env.
On Mon, Nov 17, 2014 at 4:54 AM, Tsuyoshi OZAWA <oz...@gmail.com>
wrote:
Hi Hitesh,
I'm facing with same problem in Apache version. Can you run jobs with
snappy compression with 0.5.2 release?
Thanks,
- Tsuyoshi
On Mon, Nov 17, 2014 at 9:26 PM, Hitesh Shah <hi...@apache.org> wrote:
> Hi Subroto,
>
> It could be an installer/distro issue. You may need to redirect your
question to the Hortonworks forums to get an answer on this.
>
> But yes, you are right - the env needs to be setup correctly to load the
required native libs. Out of curiosity, are you using Ambari to setup your
cluster?
>
> thanks
> — Hitesh
>
> On Nov 17, 2014, at 11:07 AM, Subroto Sanyal <sa...@gmail.com>
wrote:
>
>> Hi,
>>
>> With HDP-2.1.2 we are encountering the problem of:
>> 2014-09-11 10:14:42,881 ERROR [TezChild]
org.apache.tez.runtime.task.TezTaskRunner: Exception of type Error. Exiting
now
>> java.lang.UnsatisfiedLinkError:
org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
>> at
org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method)
>> at
org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
>> at
org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132)
>> at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>> at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>> at
org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1198)
>> at
org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1091)
>> at
org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1441)
>> at
org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:274)
>>
>>
>>
>> To workaround the problem we have to set the properties explicitly:
>> tez.am.launch.env
>>
>> tez.task.launch.env
>>
>>
>>
>> Is this a problem with HDP installation?
>>
>> --
>> Cheers,
>> Subroto Sanyal
>
--
- Tsuyoshi
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Native Compression Lib loading failed
Posted by Siddharth Seth <ss...@apache.org>.
Are these direct Tez jobs ?
The task and am env do need to be setup to make use of Snappy. The property
that needs to be set depends on the job being run. Tez direct would require
tez.am.launch.env and tez.task.launch.env to be set. For Hive, Pig, MR, etc
this may be different, depending on how these projects setup the env.
On Mon, Nov 17, 2014 at 4:54 AM, Tsuyoshi OZAWA <oz...@gmail.com>
wrote:
> Hi Hitesh,
>
> I'm facing with same problem in Apache version. Can you run jobs with
> snappy compression with 0.5.2 release?
>
> Thanks,
> - Tsuyoshi
>
>
> On Mon, Nov 17, 2014 at 9:26 PM, Hitesh Shah <hi...@apache.org> wrote:
> > Hi Subroto,
> >
> > It could be an installer/distro issue. You may need to redirect your
> question to the Hortonworks forums to get an answer on this.
> >
> > But yes, you are right - the env needs to be setup correctly to load the
> required native libs. Out of curiosity, are you using Ambari to setup your
> cluster?
> >
> > thanks
> > — Hitesh
> >
> > On Nov 17, 2014, at 11:07 AM, Subroto Sanyal <sa...@gmail.com>
> wrote:
> >
> >> Hi,
> >>
> >> With HDP-2.1.2 we are encountering the problem of:
> >> 2014-09-11 10:14:42,881 ERROR [TezChild]
> org.apache.tez.runtime.task.TezTaskRunner: Exception of type Error. Exiting
> now
> >> java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
> >> at
> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method)
> >> at
> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
> >> at
> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132)
> >> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
> >> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
> >> at
> org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1198)
> >> at
> org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1091)
> >> at
> org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1441)
> >> at
> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:274)
> >>
> >>
> >>
> >> To workaround the problem we have to set the properties explicitly:
> >> tez.am.launch.env
> >>
> >> tez.task.launch.env
> >>
> >>
> >>
> >> Is this a problem with HDP installation?
> >>
> >> --
> >> Cheers,
> >> Subroto Sanyal
> >
>
>
>
> --
> - Tsuyoshi
>
Re: Native Compression Lib loading failed
Posted by Tsuyoshi OZAWA <oz...@gmail.com>.
Hi Hitesh,
I'm facing with same problem in Apache version. Can you run jobs with
snappy compression with 0.5.2 release?
Thanks,
- Tsuyoshi
On Mon, Nov 17, 2014 at 9:26 PM, Hitesh Shah <hi...@apache.org> wrote:
> Hi Subroto,
>
> It could be an installer/distro issue. You may need to redirect your question to the Hortonworks forums to get an answer on this.
>
> But yes, you are right - the env needs to be setup correctly to load the required native libs. Out of curiosity, are you using Ambari to setup your cluster?
>
> thanks
> — Hitesh
>
> On Nov 17, 2014, at 11:07 AM, Subroto Sanyal <sa...@gmail.com> wrote:
>
>> Hi,
>>
>> With HDP-2.1.2 we are encountering the problem of:
>> 2014-09-11 10:14:42,881 ERROR [TezChild] org.apache.tez.runtime.task.TezTaskRunner: Exception of type Error. Exiting now
>> java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
>> at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method)
>> at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
>> at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132)
>> at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>> at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>> at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1198)
>> at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1091)
>> at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1441)
>> at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:274)
>>
>>
>>
>> To workaround the problem we have to set the properties explicitly:
>> tez.am.launch.env
>>
>> tez.task.launch.env
>>
>>
>>
>> Is this a problem with HDP installation?
>>
>> --
>> Cheers,
>> Subroto Sanyal
>
--
- Tsuyoshi
Re: Native Compression Lib loading failed
Posted by Hitesh Shah <hi...@apache.org>.
Hi Subroto,
It could be an installer/distro issue. You may need to redirect your question to the Hortonworks forums to get an answer on this.
But yes, you are right - the env needs to be setup correctly to load the required native libs. Out of curiosity, are you using Ambari to setup your cluster?
thanks
— Hitesh
On Nov 17, 2014, at 11:07 AM, Subroto Sanyal <sa...@gmail.com> wrote:
> Hi,
>
> With HDP-2.1.2 we are encountering the problem of:
> 2014-09-11 10:14:42,881 ERROR [TezChild] org.apache.tez.runtime.task.TezTaskRunner: Exception of type Error. Exiting now
> java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
> at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method)
> at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
> at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132)
> at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
> at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
> at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1198)
> at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1091)
> at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1441)
> at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:274)
>
>
>
> To workaround the problem we have to set the properties explicitly:
> tez.am.launch.env
>
> tez.task.launch.env
>
>
>
> Is this a problem with HDP installation?
>
> --
> Cheers,
> Subroto Sanyal